[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:11 dist-test-slave-plcg krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:11 dist-test-slave-plcg krb5kdc[395](info): set up 2 sockets
Sep 23 20:53:11 dist-test-slave-plcg krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 20:53:12.131107 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:12 dist-test-slave-plcg krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:12.309880 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:12.661326 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:12.661674 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:12.661828 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:12.670014 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:12.670135 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:12.670238 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:12.670341 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:12.670421 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:12.681082 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:12.682569 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:12.683982 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:12.697670 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:12.697613 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:12.697613 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:12.698263 408 server_base.cc:1034] running on GCE node
I20260923 20:53:12.699275 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:12.700769 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:12.701993 408 hybrid_clock.cc:648] HybridClock initialized: now 1790196792701917 us; error 73 us; skew 500 ppm
I20260923 20:53:12.702392 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:12 dist-test-slave-plcg krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:12.705646 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:12.707729 408 webserver.cc:460] Webserver started at http://127.0.97.62:43827/ using document root <none> and password file <none>
I20260923 20:53:12.708528 408 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:12.708684 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:12.709009 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:12.712205 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "2c506a429fa74772aee04defb2c6a6cb"
format_stamp: "Formatted at 2026-09-23 20:53:12 on dist-test-slave-plcg"
server_key: "913e3fd358259d92e252decde21b8f7a"
server_key_iv: "4bbb7066630b767336c1c6152ca8f999"
server_key_version: "encryptionkey@0"
I20260923 20:53:12.713032 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "2c506a429fa74772aee04defb2c6a6cb"
format_stamp: "Formatted at 2026-09-23 20:53:12 on dist-test-slave-plcg"
server_key: "913e3fd358259d92e252decde21b8f7a"
server_key_iv: "4bbb7066630b767336c1c6152ca8f999"
server_key_version: "encryptionkey@0"
I20260923 20:53:12.718899 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 20:53:12.722870 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:12.724432 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:53:12.724668 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "2c506a429fa74772aee04defb2c6a6cb"
format_stamp: "Formatted at 2026-09-23 20:53:12 on dist-test-slave-plcg"
server_key: "913e3fd358259d92e252decde21b8f7a"
server_key_iv: "4bbb7066630b767336c1c6152ca8f999"
server_key_version: "encryptionkey@0"
I20260923 20:53:12.724896 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:12.756498 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:12.761085 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:12.761416 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:12.782166 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35481
I20260923 20:53:12.782186 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35481 every 8 connection(s)
I20260923 20:53:12.783668 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:12.787827 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:12.788940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 408
I20260923 20:53:12.789247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:12.791234 388 external_mini_cluster.cc:1428] Setting key bb1415f9720fb7b8c878f4e7c831a550
I20260923 20:53:12.800840 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb: Bootstrap starting.
I20260923 20:53:12.804459 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:12.805711 475 log.cc:824] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:12.809890 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb: No bootstrap required, opened a new log
I20260923 20:53:12.818210 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c506a429fa74772aee04defb2c6a6cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260923 20:53:12.818676 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:12.818799 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c506a429fa74772aee04defb2c6a6cb, State: Initialized, Role: FOLLOWER
I20260923 20:53:12.819509 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c506a429fa74772aee04defb2c6a6cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260923 20:53:12.819777 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:12.819952 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:12.820554 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 0 FOLLOWER]: Advancing to term 1
Sep 23 20:53:12 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:12.824056 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c506a429fa74772aee04defb2c6a6cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260923 20:53:12.824752 475 leader_election.cc:302] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c506a429fa74772aee04defb2c6a6cb; no voters:
I20260923 20:53:12.825649 475 leader_election.cc:288] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:12.825856 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:12.827998 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:12.828701 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [term 1 LEADER]: Becoming Leader. State: Replica: 2c506a429fa74772aee04defb2c6a6cb, State: Running, Role: LEADER
I20260923 20:53:12.829289 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c506a429fa74772aee04defb2c6a6cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260923 20:53:12.832794 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c506a429fa74772aee04defb2c6a6cb. Latest consensus state: current_term: 1 leader_uuid: "2c506a429fa74772aee04defb2c6a6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c506a429fa74772aee04defb2c6a6cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260923 20:53:12.832748 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c506a429fa74772aee04defb2c6a6cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c506a429fa74772aee04defb2c6a6cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260923 20:53:12.833225 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:12.833271 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:12.837743 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:12.795295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.0.97.62:35481)
0923 20:53:12.796125 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:12.796138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:12.797589 (+ 1451us) server_negotiation.cc:407] Connection header received
0923 20:53:12.807557 (+ 9968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:12.807586 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:12.808046 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:12.808529 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:12.811956 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:12.813438 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:12.815317 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:12.816064 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:12.816299 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:12.823458 (+ 7159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:12.823468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:12.823485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:12.823534 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:12.828157 (+ 4623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:12.829505 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:12.829514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:12.829523 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:12.829765 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:12.830610 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:12.830615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:12.830618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:12.831529 (+ 911us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:12.832423 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:12.835141 (+ 2718us) server_negotiation.cc:299] Negotiation successful
0923 20:53:12.835784 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":373,"threads_started":1}
I20260923 20:53:12.843670 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:12.851102 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:12.863376 485 catalog_manager.cc:1347] Generated new cluster ID: 155adfb5885347209f2ecf9b3998888b
I20260923 20:53:12.863529 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:12.877800 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:12.879634 485 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:12.887135 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c506a429fa74772aee04defb2c6a6cb: Generated new TSK 0
I20260923 20:53:12.888010 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:12.938876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:13.293707 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:13.294106 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:13.294277 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:13.301663 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:13.301852 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:13.302049 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:13.312711 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:13.314307 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:13.316071 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:13.330446 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:13.330530 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:13.330472 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:13.331439 501 server_base.cc:1034] running on GCE node
I20260923 20:53:13.332172 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:13.333236 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:13.334496 501 hybrid_clock.cc:648] HybridClock initialized: now 1790196793334432 us; error 51 us; skew 500 ppm
I20260923 20:53:13.334867 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:13 dist-test-slave-plcg krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:13.338301 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:13.340329 501 webserver.cc:460] Webserver started at http://127.0.97.1:33115/ using document root <none> and password file <none>
I20260923 20:53:13.341112 501 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:13.341250 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:13.341601 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:13.344765 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "8287d40d35a946d88c13e21aa98050e9"
format_stamp: "Formatted at 2026-09-23 20:53:13 on dist-test-slave-plcg"
server_key: "83de0d25a8c0d4d7cf009c5495c5a662"
server_key_iv: "de2e8a80dc7ae3656d450edb6060e983"
server_key_version: "encryptionkey@0"
I20260923 20:53:13.345669 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8287d40d35a946d88c13e21aa98050e9"
format_stamp: "Formatted at 2026-09-23 20:53:13 on dist-test-slave-plcg"
server_key: "83de0d25a8c0d4d7cf009c5495c5a662"
server_key_iv: "de2e8a80dc7ae3656d450edb6060e983"
server_key_version: "encryptionkey@0"
I20260923 20:53:13.351858 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:53:13.355654 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:13.357023 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:53:13.357197 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "8287d40d35a946d88c13e21aa98050e9"
format_stamp: "Formatted at 2026-09-23 20:53:13 on dist-test-slave-plcg"
server_key: "83de0d25a8c0d4d7cf009c5495c5a662"
server_key_iv: "de2e8a80dc7ae3656d450edb6060e983"
server_key_version: "encryptionkey@0"
I20260923 20:53:13.357414 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:13.385283 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:13.390000 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:13.390311 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:13.391323 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:13.392729 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:13.392807 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:13.392980 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:13.393083 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:13.430711 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38067
I20260923 20:53:13.430749 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38067 every 8 connection(s)
I20260923 20:53:13.432080 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:13.439359 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 501
I20260923 20:53:13.439581 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:13.440346 388 external_mini_cluster.cc:1428] Setting key a9f4270f82eafefde52ab67ebfef8c48
Sep 23 20:53:13 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:13.455613 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:13.434886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42333 (local address 127.0.97.62:35481)
0923 20:53:13.435313 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:13.435317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:13.436420 (+ 1103us) server_negotiation.cc:407] Connection header received
0923 20:53:13.438326 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:13.438330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:13.438410 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:13.438511 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:13.440630 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.441349 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.443020 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.443427 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.443566 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:13.447642 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:13.447652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:13.447660 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:13.447709 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:13.451907 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:13.452803 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:13.452811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:13.452818 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:13.452978 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:13.453534 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:13.453539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:13.453542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:13.453966 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:13.454133 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:13.454881 (+ 748us) server_negotiation.cc:299] Negotiation successful
0923 20:53:13.455158 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":210,"threads_started":1}
I20260923 20:53:13.456331 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:13.435496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.1:42333)
0923 20:53:13.436186 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:13.436267 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:13.438050 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:13.438735 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:13.438744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:13.439566 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:13.440426 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:13.440457 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.441534 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.441543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:13.442853 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:13.442867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.443684 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.443691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:13.443945 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:13.444678 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:13.444702 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:13.447357 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:13.452117 (+ 4760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:13.452126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:13.452143 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:13.452639 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:13.453095 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:13.453098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:13.453101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:13.453380 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:13.454093 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:13.454100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:13.454452 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 20:53:13.454694 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 20:53:13.455114 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":276,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:53:13.458249 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260923 20:53:13.458546 629 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:13.459362 629 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 20:53:13.462170 440 ts_manager.cc:194] Registered new tserver with Master: 8287d40d35a946d88c13e21aa98050e9 (127.0.97.1:38067)
I20260923 20:53:13.464926 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:13.484722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:13.838162 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:13.838508 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:13.839097 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:13.846678 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:13.846849 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:13.847024 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:13.857323 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:13.858748 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:13.860208 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:13.874434 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:13.874709 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:13.874996 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:13.875905 635 server_base.cc:1034] running on GCE node
I20260923 20:53:13.876525 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:13.877458 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:13.878655 635 hybrid_clock.cc:648] HybridClock initialized: now 1790196793878611 us; error 35 us; skew 500 ppm
I20260923 20:53:13.878949 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:13 dist-test-slave-plcg krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:13.882441 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:13.889618 635 webserver.cc:460] Webserver started at http://127.0.97.2:32781/ using document root <none> and password file <none>
I20260923 20:53:13.890497 635 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:13.890669 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:13.891121 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:13.894866 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "559f9ae596d24b6a96e7132571dd2a40"
format_stamp: "Formatted at 2026-09-23 20:53:13 on dist-test-slave-plcg"
server_key: "790c42bcba4034c172279abc2f030c7d"
server_key_iv: "516f6fe7349809422c07f956aba567a7"
server_key_version: "encryptionkey@0"
I20260923 20:53:13.895830 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "559f9ae596d24b6a96e7132571dd2a40"
format_stamp: "Formatted at 2026-09-23 20:53:13 on dist-test-slave-plcg"
server_key: "790c42bcba4034c172279abc2f030c7d"
server_key_iv: "516f6fe7349809422c07f956aba567a7"
server_key_version: "encryptionkey@0"
I20260923 20:53:13.901919 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:53:13.905741 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:13.907280 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:53:13.907483 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "559f9ae596d24b6a96e7132571dd2a40"
format_stamp: "Formatted at 2026-09-23 20:53:13 on dist-test-slave-plcg"
server_key: "790c42bcba4034c172279abc2f030c7d"
server_key_iv: "516f6fe7349809422c07f956aba567a7"
server_key_version: "encryptionkey@0"
I20260923 20:53:13.907711 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:13.926962 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:13.932098 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:13.932402 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:13.933441 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:13.934963 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:13.935073 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:13.935238 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:13.935313 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:13.972522 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42493
I20260923 20:53:13.972535 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42493 every 8 connection(s)
I20260923 20:53:13.973953 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:13.983045 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 635
I20260923 20:53:13.983263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:13.983824 388 external_mini_cluster.cc:1428] Setting key 53266896906a1eeb580db09605292657
Sep 23 20:53:13 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:13.998973 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:13.976710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42381 (local address 127.0.97.62:35481)
0923 20:53:13.977204 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:13.977211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:13.978437 (+ 1226us) server_negotiation.cc:407] Connection header received
0923 20:53:13.980789 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:13.980794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:13.980870 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:13.980998 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:13.982764 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.984028 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.985988 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.986436 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.986583 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:13.990599 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:13.990609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:13.990616 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:13.990683 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:13.994626 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:13.995914 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:13.995923 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:13.995930 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:13.996129 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:13.996629 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:13.996637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:13.996643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:13.997084 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:13.997253 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:13.998190 (+ 937us) server_negotiation.cc:299] Negotiation successful
0923 20:53:13.998482 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":230,"threads_started":1}
I20260923 20:53:13.999643 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:13.977448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.2:42381)
0923 20:53:13.978252 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:13.978303 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:13.980450 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:13.981228 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:13.981237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:13.981891 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:13.982532 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:13.982551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.984198 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.984202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:13.985806 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:13.985848 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:13.986701 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:13.986709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:13.986945 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:13.987694 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:13.987724 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:13.990269 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:13.994901 (+ 4632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:13.994912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:13.994933 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:13.995716 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:13.996267 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:13.996271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:13.996274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:13.996483 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:13.997243 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:13.997251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:13.997657 (+ 406us) client_negotiation.cc:769] Sending connection context
0923 20:53:13.998038 (+ 381us) client_negotiation.cc:240] Negotiation successful
0923 20:53:13.998497 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:53:14.001576 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260923 20:53:14.001936 763 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:14.003180 763 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 20:53:14.004877 440 ts_manager.cc:194] Registered new tserver with Master: 559f9ae596d24b6a96e7132571dd2a40 (127.0.97.2:42493)
I20260923 20:53:14.005733 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:14.026733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:14.370927 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:14.371294 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:14.371480 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:14.378850 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:14.379025 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:14.379216 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:14.389084 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:14.390509 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:14.391907 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:14.406177 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:14.406299 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:14.406427 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:14.407835 769 server_base.cc:1034] running on GCE node
I20260923 20:53:14.408587 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:14.409624 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:14.410910 769 hybrid_clock.cc:648] HybridClock initialized: now 1790196794410856 us; error 59 us; skew 500 ppm
I20260923 20:53:14.411319 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:14.415266 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:14.417047 769 webserver.cc:460] Webserver started at http://127.0.97.3:40513/ using document root <none> and password file <none>
I20260923 20:53:14.417801 769 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:14.418013 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:14.418346 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:14.421438 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "c9875a02517b498cb2470ac6265ef5f1"
format_stamp: "Formatted at 2026-09-23 20:53:14 on dist-test-slave-plcg"
server_key: "48896fd7bb95c9ad2f75c87d8e734590"
server_key_iv: "789488f0e2e5d078010d7d8673f23ff6"
server_key_version: "encryptionkey@0"
I20260923 20:53:14.422359 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c9875a02517b498cb2470ac6265ef5f1"
format_stamp: "Formatted at 2026-09-23 20:53:14 on dist-test-slave-plcg"
server_key: "48896fd7bb95c9ad2f75c87d8e734590"
server_key_iv: "789488f0e2e5d078010d7d8673f23ff6"
server_key_version: "encryptionkey@0"
I20260923 20:53:14.427790 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:53:14.431980 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:14.433542 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:53:14.433740 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "c9875a02517b498cb2470ac6265ef5f1"
format_stamp: "Formatted at 2026-09-23 20:53:14 on dist-test-slave-plcg"
server_key: "48896fd7bb95c9ad2f75c87d8e734590"
server_key_iv: "789488f0e2e5d078010d7d8673f23ff6"
server_key_version: "encryptionkey@0"
I20260923 20:53:14.434057 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:14.461345 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:14.466035 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:14.466353 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:14.467362 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:14.468513 629 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
I20260923 20:53:14.468852 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:14.468962 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:14.469089 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:14.469161 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:14.506794 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44789
I20260923 20:53:14.506848 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44789 every 8 connection(s)
I20260923 20:53:14.508196 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:14.514621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 769
I20260923 20:53:14.514832 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:14.515372 388 external_mini_cluster.cc:1428] Setting key 62a345fd91bfe387055fe257a4596fba
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:14.531258 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:14.510994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49025 (local address 127.0.97.62:35481)
0923 20:53:14.511470 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:14.511476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:14.512789 (+ 1313us) server_negotiation.cc:407] Connection header received
0923 20:53:14.514948 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:14.514961 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:14.515038 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:14.515150 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:14.517381 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.518438 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.520300 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.520692 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.520830 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:14.524452 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:14.524461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:14.524464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:14.524510 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:14.527647 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.528469 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.528473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.528476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.528638 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.529124 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.529133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.529135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.529397 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:14.529575 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:14.530537 (+ 962us) server_negotiation.cc:299] Negotiation successful
0923 20:53:14.530842 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260923 20:53:14.532457 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:14.511726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.3:49025)
0923 20:53:14.512509 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:14.512552 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:14.514537 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:14.515474 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:14.515488 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:14.516321 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:14.517151 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:14.517174 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.518617 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.518623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:14.520133 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:14.520151 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.520926 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.520935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:14.521201 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:14.521950 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:14.521977 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:14.524208 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:14.527826 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:14.527834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:14.527850 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:14.528340 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:14.528763 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:14.528772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:14.528775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:14.528988 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:14.529559 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:14.529573 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:14.530119 (+ 546us) client_negotiation.cc:769] Sending connection context
0923 20:53:14.530414 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 20:53:14.531005 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260923 20:53:14.534085 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260923 20:53:14.534435 897 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:14.535215 897 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260923 20:53:14.536828 440 ts_manager.cc:194] Registered new tserver with Master: c9875a02517b498cb2470ac6265ef5f1 (127.0.97.3:44789)
I20260923 20:53:14.537698 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49025
I20260923 20:53:14.544644 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:14.591593 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:14.574850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.0.97.1:38067)
0923 20:53:14.575551 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:14.575562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:14.575666 (+ 104us) server_negotiation.cc:407] Connection header received
0923 20:53:14.575904 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:14.575911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:14.576152 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:14.576327 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:14.578321 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.579350 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.580745 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.581331 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.581486 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:14.585671 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:14.585686 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:14.585702 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:14.585741 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:14.588589 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.589450 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.589459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.589461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.589610 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.590190 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.590195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.590197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.590575 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:14.590832 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:14.591087 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 20:53:14.591298 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":334,"threads_started":1}
W20260923 20:53:14.592905 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45464
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:14.637086 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:14.623422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.97.1:38067)
0923 20:53:14.623671 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:14.623677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:14.623943 (+ 266us) server_negotiation.cc:407] Connection header received
0923 20:53:14.624085 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:14.624089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:14.624186 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:14.624274 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:14.625546 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.626336 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.627514 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.628002 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.628097 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:14.632280 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:14.632285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:14.632288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:14.632325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:14.634790 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.635440 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.635445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.635447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.635569 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.635886 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.635889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.635891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.636150 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:14.636317 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:14.636557 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 20:53:14.636716 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:53:14.638804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 501
I20260923 20:53:14.651609 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:14.639807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.0.97.1:38067)
0923 20:53:14.639970 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:14.639989 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:14.640363 (+ 374us) server_negotiation.cc:407] Connection header received
0923 20:53:14.640477 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:14.640482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:14.640534 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:14.640611 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:14.641720 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.642447 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.643637 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.644080 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.644181 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:14.646551 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:14.646556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:14.646559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:14.646592 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:14.649211 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.649918 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.649922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.649925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.650079 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.650406 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.650410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.650412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.650682 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:14.650861 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:14.651070 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 20:53:14.651327 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:53:14.652303 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:14.739178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 501
I20260923 20:53:14.755870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 635
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:53:14.772099 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:14.756731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.0.97.2:42493)
0923 20:53:14.757314 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:14.757321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:14.757347 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:53:14.757412 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:14.757419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:14.757634 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:14.757771 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:14.758967 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.759990 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.761239 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.761807 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.761985 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:14.766326 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:14.766341 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:14.766358 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:14.766406 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:14.769413 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.770232 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.770240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.770242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.770370 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.770740 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.770743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.770745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.771081 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:14.771300 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:14.771580 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:53:14.771783 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":316,"threads_started":1}
I20260923 20:53:14.773407 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:14.862804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 635
I20260923 20:53:14.885582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 769
Sep 23 20:53:14 dist-test-slave-plcg krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:53:14.902096 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:14.886627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.0.97.3:44789)
0923 20:53:14.887212 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:14.887218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:14.887246 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:53:14.887303 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:14.887309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:14.887520 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:14.887656 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:14.888865 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.889975 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.891221 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:14.891804 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:14.891913 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:14.896312 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:14.896327 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:14.896344 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:14.896385 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:14.899533 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.900238 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.900245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.900247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.900363 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:14.900669 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:14.900672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:14.900674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:14.901006 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:14.901236 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:14.901462 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 20:53:14.901668 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":290,"threads_started":1}
I20260923 20:53:14.903384 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:14.986307 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 769
I20260923 20:53:15.004294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 408
I20260923 20:53:15.005650 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:15.107928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 408
2026-09-23T20:53:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4078 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:15 dist-test-slave-plcg krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:15 dist-test-slave-plcg krb5kdc[919](info): set up 2 sockets
Sep 23 20:53:15 dist-test-slave-plcg krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260923 20:53:16.197805 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:16 dist-test-slave-plcg krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:16.381145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38221
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:16.718391 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:16.718713 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:16.718883 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:16.726286 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:16.726411 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:16.726501 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:16.726616 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:16.726693 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:16.736843 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:16.738436 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:16.739861 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:16.752503 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:16.752503 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:16.752975 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:16.753479 932 server_base.cc:1034] running on GCE node
I20260923 20:53:16.754424 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:16.756088 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:16.757300 932 hybrid_clock.cc:648] HybridClock initialized: now 1790196796757221 us; error 74 us; skew 500 ppm
I20260923 20:53:16.757643 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:16 dist-test-slave-plcg krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:16.773160 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:16.775239 932 webserver.cc:460] Webserver started at http://127.0.97.62:37583/ using document root <none> and password file <none>
I20260923 20:53:16.775991 932 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:16.776101 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:16.776368 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:16.780038 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "9559d814e6d44bfc9950ab3bb50b5ce6"
format_stamp: "Formatted at 2026-09-23 20:53:16 on dist-test-slave-plcg"
server_key: "0d609e1eedd48f1c0b611a8f04d8ab5b"
server_key_iv: "6f5fe24b0e84bf2800098f1ae2f64f1d"
server_key_version: "encryptionkey@0"
I20260923 20:53:16.780827 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "9559d814e6d44bfc9950ab3bb50b5ce6"
format_stamp: "Formatted at 2026-09-23 20:53:16 on dist-test-slave-plcg"
server_key: "0d609e1eedd48f1c0b611a8f04d8ab5b"
server_key_iv: "6f5fe24b0e84bf2800098f1ae2f64f1d"
server_key_version: "encryptionkey@0"
I20260923 20:53:16.787058 932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:53:16.791606 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:16.793157 932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:53:16.793334 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "9559d814e6d44bfc9950ab3bb50b5ce6"
format_stamp: "Formatted at 2026-09-23 20:53:16 on dist-test-slave-plcg"
server_key: "0d609e1eedd48f1c0b611a8f04d8ab5b"
server_key_iv: "6f5fe24b0e84bf2800098f1ae2f64f1d"
server_key_version: "encryptionkey@0"
I20260923 20:53:16.793527 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:16.811681 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:16.816354 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:16.816619 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:16.836014 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38221
I20260923 20:53:16.836064 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38221 every 8 connection(s)
I20260923 20:53:16.837538 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:16.840003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 932
I20260923 20:53:16.840240 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:16.840916 388 external_mini_cluster.cc:1428] Setting key 274ab434c7fea536214b30a52ef28171
I20260923 20:53:16.841953 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:53:16 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:16.856835 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6: Bootstrap starting.
I20260923 20:53:16.861260 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:16.862789 999 log.cc:824] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:16.866600 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6: No bootstrap required, opened a new log
I20260923 20:53:16.867781 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:16.843137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.97.62:38221)
0923 20:53:16.843973 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:16.843991 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:16.844046 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:53:16.845891 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:16.845924 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:16.846540 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:16.847059 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:16.848388 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:16.850555 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:16.851904 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:16.852981 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:16.853200 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:16.858260 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:16.858273 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:16.858296 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:16.858349 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:16.862601 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:16.863625 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:16.863633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:16.863640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:16.863834 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:16.864304 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:16.864309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:16.864311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:16.865078 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:16.865371 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:16.865855 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 20:53:16.866357 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260923 20:53:16.875341 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38221 } }
I20260923 20:53:16.875892 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:16.876026 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9559d814e6d44bfc9950ab3bb50b5ce6, State: Initialized, Role: FOLLOWER
I20260923 20:53:16.876610 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38221 } }
I20260923 20:53:16.876850 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:16.876991 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:16.877169 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:16.879513 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38221 } }
I20260923 20:53:16.880060 999 leader_election.cc:302] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9559d814e6d44bfc9950ab3bb50b5ce6; no voters:
I20260923 20:53:16.880682 999 leader_election.cc:288] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:16.880846 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:16.881592 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 9559d814e6d44bfc9950ab3bb50b5ce6, State: Running, Role: LEADER
I20260923 20:53:16.882174 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38221 } }
I20260923 20:53:16.883056 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:16.885530 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38221 } } }
I20260923 20:53:16.885488 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9559d814e6d44bfc9950ab3bb50b5ce6. Latest consensus state: current_term: 1 leader_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9559d814e6d44bfc9950ab3bb50b5ce6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38221 } } }
I20260923 20:53:16.886238 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:16.886262 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:16.887266 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:16.894567 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:16.906076 1012 catalog_manager.cc:1347] Generated new cluster ID: 0fd07240b8c24a979963695f8b90adc1
I20260923 20:53:16.906255 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:16.924551 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:16.926368 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:16.934288 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9559d814e6d44bfc9950ab3bb50b5ce6: Generated new TSK 0
I20260923 20:53:16.935250 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:16.979863 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38221
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:17.323863 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:17.324169 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:17.324268 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:17.332094 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:17.332226 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:17.332341 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:17.342682 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38221
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:17.343986 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:17.345300 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:17.359151 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:17.359156 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:17.359664 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:17.360049 1025 server_base.cc:1034] running on GCE node
I20260923 20:53:17.360729 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:17.361706 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:17.362948 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790196797362896 us; error 51 us; skew 500 ppm
I20260923 20:53:17.363284 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:17 dist-test-slave-plcg krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:17.366617 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:17.368149 1025 webserver.cc:460] Webserver started at http://127.0.97.1:38773/ using document root <none> and password file <none>
I20260923 20:53:17.368897 1025 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:17.369058 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:17.369387 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:17.372548 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "3049ba088cee41cb9e64f3421cdbc7ca"
format_stamp: "Formatted at 2026-09-23 20:53:17 on dist-test-slave-plcg"
server_key: "61235c81c770632384239dc1fdd97940"
server_key_iv: "46df857d65b3433c265def2bc08c4fea"
server_key_version: "encryptionkey@0"
I20260923 20:53:17.373404 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3049ba088cee41cb9e64f3421cdbc7ca"
format_stamp: "Formatted at 2026-09-23 20:53:17 on dist-test-slave-plcg"
server_key: "61235c81c770632384239dc1fdd97940"
server_key_iv: "46df857d65b3433c265def2bc08c4fea"
server_key_version: "encryptionkey@0"
I20260923 20:53:17.379029 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:53:17.383004 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:17.384626 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:53:17.384828 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "3049ba088cee41cb9e64f3421cdbc7ca"
format_stamp: "Formatted at 2026-09-23 20:53:17 on dist-test-slave-plcg"
server_key: "61235c81c770632384239dc1fdd97940"
server_key_iv: "46df857d65b3433c265def2bc08c4fea"
server_key_version: "encryptionkey@0"
I20260923 20:53:17.385071 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:17.413280 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:17.417876 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:17.418196 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:17.419239 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:17.420701 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:17.420812 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:17.420965 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:17.421041 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:17.455935 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40879
I20260923 20:53:17.455981 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40879 every 8 connection(s)
I20260923 20:53:17.457333 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:17.458539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 1025
I20260923 20:53:17.458767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:17.459540 388 external_mini_cluster.cc:1428] Setting key 4b0976abed5a4909ae09b7ebd7f3536a
Sep 23 20:53:17 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 20:53:17.479920 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:17.460339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46809 (local address 127.0.97.62:38221)
0923 20:53:17.460805 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:17.460812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:17.461922 (+ 1110us) server_negotiation.cc:407] Connection header received
0923 20:53:17.464947 (+ 3025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:17.464952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:17.465024 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:17.465132 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:17.467454 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.468157 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.469775 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.470247 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.470384 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:17.473981 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:17.473986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:17.473989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:17.474022 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:17.476761 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:17.477564 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:17.477569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:17.477571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:17.477767 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:17.478218 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:17.478222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:17.478224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:17.478483 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:17.478604 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:17.479296 (+ 692us) server_negotiation.cc:299] Negotiation successful
0923 20:53:17.479552 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260923 20:53:17.481000 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:17.460855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38221 (local address 127.0.97.1:46809)
0923 20:53:17.461717 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:17.461779 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:17.464559 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:17.465450 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:17.465466 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:17.466365 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:17.467256 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:17.467278 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.468331 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.468336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:17.469604 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:17.469617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.470489 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.470496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:17.470761 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:17.471502 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:17.471525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:17.473698 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:17.476951 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:17.476959 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:17.476975 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:17.477428 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:17.477911 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:17.477914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:17.477916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:17.478112 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:17.478604 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:17.478611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:17.478947 (+ 336us) client_negotiation.cc:769] Sending connection context
0923 20:53:17.479179 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 20:53:17.479713 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":312,"threads_started":1}
I20260923 20:53:17.482723 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38221
I20260923 20:53:17.483089 1153 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:17.483994 1153 heartbeater.cc:511] Master 127.0.97.62:38221 requested a full tablet report, sending...
I20260923 20:53:17.486526 964 ts_manager.cc:194] Registered new tserver with Master: 3049ba088cee41cb9e64f3421cdbc7ca (127.0.97.1:40879)
I20260923 20:53:17.488785 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:17.504696 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38221
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:17.839359 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:17.839780 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:17.839929 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:17.851681 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:17.851826 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:17.852016 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:17.864827 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38221
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:17.866230 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:17.867590 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:17.881608 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:17.881659 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:17.881561 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:17.893096 1159 server_base.cc:1034] running on GCE node
I20260923 20:53:17.894003 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:17.895185 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:17.896441 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790196797896387 us; error 63 us; skew 500 ppm
I20260923 20:53:17.896828 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:17 dist-test-slave-plcg krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:17.900180 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:17.901764 1159 webserver.cc:460] Webserver started at http://127.0.97.2:36623/ using document root <none> and password file <none>
I20260923 20:53:17.902557 1159 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:17.902709 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:17.903041 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:17.906220 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "4564766227384329a9c22b69e577b812"
format_stamp: "Formatted at 2026-09-23 20:53:17 on dist-test-slave-plcg"
server_key: "8f6461dc36d24dd40be2b87baa6843fc"
server_key_iv: "170f245137598ee8455f6005b069addd"
server_key_version: "encryptionkey@0"
I20260923 20:53:17.907048 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4564766227384329a9c22b69e577b812"
format_stamp: "Formatted at 2026-09-23 20:53:17 on dist-test-slave-plcg"
server_key: "8f6461dc36d24dd40be2b87baa6843fc"
server_key_iv: "170f245137598ee8455f6005b069addd"
server_key_version: "encryptionkey@0"
I20260923 20:53:17.912714 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:53:17.916546 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:17.918155 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:53:17.918347 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "4564766227384329a9c22b69e577b812"
format_stamp: "Formatted at 2026-09-23 20:53:17 on dist-test-slave-plcg"
server_key: "8f6461dc36d24dd40be2b87baa6843fc"
server_key_iv: "170f245137598ee8455f6005b069addd"
server_key_version: "encryptionkey@0"
I20260923 20:53:17.918568 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:17.932633 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:17.937561 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:17.937999 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:17.939383 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:17.940932 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:17.941052 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:17.941219 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:17.941293 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:17.977183 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45971
I20260923 20:53:17.977243 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45971 every 8 connection(s)
I20260923 20:53:17.978652 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:17.983464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 1159
I20260923 20:53:17.983682 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:17.984261 388 external_mini_cluster.cc:1428] Setting key a54e4bf61cf867fe21c89251804269d6
Sep 23 20:53:17 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:53:18.004294 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:17.981463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51583 (local address 127.0.97.62:38221)
0923 20:53:17.981996 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:17.982002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:17.983214 (+ 1212us) server_negotiation.cc:407] Connection header received
0923 20:53:17.985783 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:17.985787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:17.985920 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:17.986018 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:17.988442 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.989311 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.991952 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.992732 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.992900 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:17.996657 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:17.996668 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:17.996676 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:17.996728 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:18.000712 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.001630 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.001636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.001640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.001891 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.002351 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.002359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.002364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.002774 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:18.002889 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.003789 (+ 900us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.003975 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":225,"threads_started":1}
I20260923 20:53:18.005477 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:17.982181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38221 (local address 127.0.97.2:51583)
0923 20:53:17.982960 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:17.983058 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:17.985423 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:17.986347 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:17.986358 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:17.987188 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:17.988199 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:17.988232 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.989528 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.989537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:17.991745 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:17.991760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:17.993012 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:17.993018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:17.993249 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:17.994083 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:17.994114 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:17.996339 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:18.000928 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:18.000938 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:18.000954 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:18.001449 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:18.002016 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:18.002020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:18.002023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:18.002218 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:18.002941 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:18.002948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:18.003349 (+ 401us) client_negotiation.cc:769] Sending connection context
0923 20:53:18.003669 (+ 320us) client_negotiation.cc:240] Negotiation successful
0923 20:53:18.004125 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":320,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 20:53:18.007261 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38221
I20260923 20:53:18.007606 1287 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:18.008397 1287 heartbeater.cc:511] Master 127.0.97.62:38221 requested a full tablet report, sending...
I20260923 20:53:18.010610 964 ts_manager.cc:194] Registered new tserver with Master: 4564766227384329a9c22b69e577b812 (127.0.97.2:45971)
I20260923 20:53:18.011821 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:18.030299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38221
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:18.377312 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:18.377610 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:18.377765 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:18.385342 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:18.385469 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:18.385592 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:18.396128 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38221
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:18.397559 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:18.398979 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:18.413031 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:18.413031 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:18.413071 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:18.413959 1293 server_base.cc:1034] running on GCE node
I20260923 20:53:18.414567 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:18.415586 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:18.416838 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790196798416795 us; error 34 us; skew 500 ppm
I20260923 20:53:18.417183 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:18 dist-test-slave-plcg krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:18.420518 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:18.422278 1293 webserver.cc:460] Webserver started at http://127.0.97.3:45505/ using document root <none> and password file <none>
I20260923 20:53:18.423051 1293 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:18.423233 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:18.423571 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:18.426792 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "dc1990a7933d48eda82f1919e8eed1bf"
format_stamp: "Formatted at 2026-09-23 20:53:18 on dist-test-slave-plcg"
server_key: "37b99235ff5898ee04445aee977cbe0a"
server_key_iv: "861578acd2880e57bf54395510f98d79"
server_key_version: "encryptionkey@0"
I20260923 20:53:18.427660 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dc1990a7933d48eda82f1919e8eed1bf"
format_stamp: "Formatted at 2026-09-23 20:53:18 on dist-test-slave-plcg"
server_key: "37b99235ff5898ee04445aee977cbe0a"
server_key_iv: "861578acd2880e57bf54395510f98d79"
server_key_version: "encryptionkey@0"
I20260923 20:53:18.433804 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 20:53:18.437685 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:18.439261 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:53:18.439471 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "dc1990a7933d48eda82f1919e8eed1bf"
format_stamp: "Formatted at 2026-09-23 20:53:18 on dist-test-slave-plcg"
server_key: "37b99235ff5898ee04445aee977cbe0a"
server_key_iv: "861578acd2880e57bf54395510f98d79"
server_key_version: "encryptionkey@0"
I20260923 20:53:18.439693 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:18.457549 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:18.462512 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:18.462841 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:18.463905 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:18.465340 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:18.465476 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:18.465639 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:18.465718 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:18.492506 1153 heartbeater.cc:503] Master 127.0.97.62:38221 was elected leader, sending a full tablet report...
I20260923 20:53:18.501710 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39055
I20260923 20:53:18.501752 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39055 every 8 connection(s)
I20260923 20:53:18.503063 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:18.509083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 1293
I20260923 20:53:18.509291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:18.509698 388 external_mini_cluster.cc:1428] Setting key 1d93b81fd572b2c42e6e70c4bd569420
Sep 23 20:53:18 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:18.524632 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.505598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45735 (local address 127.0.97.62:38221)
0923 20:53:18.506072 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.506077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.507046 (+ 969us) server_negotiation.cc:407] Connection header received
0923 20:53:18.508961 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.508966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.509083 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.509218 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:18.511297 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.512195 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.514068 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.514510 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.514624 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.518349 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:18.518354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:18.518358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:18.518394 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:18.521055 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.521946 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.521951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.521953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.522081 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.522601 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.522605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.522607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.522874 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:18.523019 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.523988 (+ 969us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.524241 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":268,"threads_started":1}
I20260923 20:53:18.525738 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.506173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38221 (local address 127.0.97.3:45735)
0923 20:53:18.506867 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:18.506908 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:18.508637 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:18.509536 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:18.509552 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:18.510218 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:18.511083 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.511120 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.512361 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.512366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.513897 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.513911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.514731 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.514737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.514950 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.515694 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:18.515722 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:18.518091 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:18.521225 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:18.521233 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:18.521250 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:18.521721 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:18.522189 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:18.522192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:18.522195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:18.522455 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:18.523019 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:18.523025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:18.523557 (+ 532us) client_negotiation.cc:769] Sending connection context
0923 20:53:18.523862 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 20:53:18.524420 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":254,"threads_started":1}
I20260923 20:53:18.527273 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38221
I20260923 20:53:18.527576 1421 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:18.528273 1421 heartbeater.cc:511] Master 127.0.97.62:38221 requested a full tablet report, sending...
I20260923 20:53:18.529735 964 ts_manager.cc:194] Registered new tserver with Master: dc1990a7933d48eda82f1919e8eed1bf (127.0.97.3:39055)
I20260923 20:53:18.530609 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45735
I20260923 20:53:18.538375 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:53:18.559443 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.545215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.0.97.62:38221)
0923 20:53:18.545408 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.545412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.546017 (+ 605us) server_negotiation.cc:407] Connection header received
0923 20:53:18.546251 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.546255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.546323 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.546427 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:18.548415 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.549186 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.550765 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.551220 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.551366 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.553981 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:18.553987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:18.553990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:18.554030 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:18.556618 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.557352 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.557358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.557363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.557486 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.557946 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.557951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.557953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.558277 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:18.558524 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.558881 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.559113 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:53:18.566093 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:18.573530 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:18.597694 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.586534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40879 (local address 127.0.0.1:39004)
0923 20:53:18.588454 (+ 1920us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:18.588504 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:18.588888 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:18.590246 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:18.590255 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:18.590293 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:18.590708 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.590721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.592661 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.592669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.595250 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.595261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.596642 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.596648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.596847 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.596922 (+ 75us) client_negotiation.cc:769] Sending connection context
0923 20:53:18.597083 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 20:53:18.597304 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"thread_start_us":2302,"threads_started":1}
I20260923 20:53:18.597688 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.586683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.0.97.1:40879)
0923 20:53:18.587351 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.587360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.588790 (+ 1430us) server_negotiation.cc:407] Connection header received
0923 20:53:18.589298 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.589311 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.589682 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.590017 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:18.590981 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.592519 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.595421 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.596406 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.596573 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.596833 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.597115 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.597327 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":691,"threads_started":1}
I20260923 20:53:18.601120 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.589559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.0.97.3:39055)
0923 20:53:18.590813 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.590826 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.591381 (+ 555us) server_negotiation.cc:407] Connection header received
0923 20:53:18.591648 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.591657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.591904 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.592102 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:18.592989 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.594322 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.597544 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.599507 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.599612 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.599821 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.600606 (+ 785us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.600780 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":512,"threads_started":1}
I20260923 20:53:18.601258 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.589947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39055 (local address 127.0.0.1:47100)
0923 20:53:18.591264 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:18.591285 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:18.591484 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:18.592310 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:18.592316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:18.592348 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:18.592823 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.592832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.594517 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.594523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.597347 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.597362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.600042 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.600047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.600302 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.600421 (+ 119us) client_negotiation.cc:769] Sending connection context
0923 20:53:18.600521 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 20:53:18.600776 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":337,"thread_start_us":1079,"threads_started":1}
I20260923 20:53:18.601310 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.587233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45971 (local address 127.0.0.1:48198)
0923 20:53:18.589143 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:18.589162 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:18.589273 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:18.592057 (+ 2784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:18.592061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:18.592087 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:18.592514 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.592527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.594501 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.594506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.597347 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.597362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.600006 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.600023 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.600302 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.600423 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 20:53:18.600526 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:53:18.600776 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"mutex_wait_us":381,"thread_start_us":659,"threads_started":1}
I20260923 20:53:18.601948 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.587626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.0.97.2:45971)
0923 20:53:18.591567 (+ 3941us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.591576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.591603 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:53:18.591688 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.591692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.591874 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.592070 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:18.592740 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.594346 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.597555 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.599772 (+ 2217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.599907 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.600093 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.601455 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.601646 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3768,"thread_start_us":3658,"threads_started":1}
I20260923 20:53:18.602654 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0fc1b8aa8d496b861f2fc3e7fc92c0 (DEFAULT_TABLE table=test-table [id=813b534a9e114e3bb2759daa9a11521c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:18.604101 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0fc1b8aa8d496b861f2fc3e7fc92c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:18.605124 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0fc1b8aa8d496b861f2fc3e7fc92c0 (DEFAULT_TABLE table=test-table [id=813b534a9e114e3bb2759daa9a11521c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:18.605370 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0fc1b8aa8d496b861f2fc3e7fc92c0 (DEFAULT_TABLE table=test-table [id=813b534a9e114e3bb2759daa9a11521c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:18.606652 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0fc1b8aa8d496b861f2fc3e7fc92c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:18.606662 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0fc1b8aa8d496b861f2fc3e7fc92c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:18.617100 1440 tablet_bootstrap.cc:492] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca: Bootstrap starting.
I20260923 20:53:18.621897 1440 tablet_bootstrap.cc:654] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:18.622668 1441 tablet_bootstrap.cc:492] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf: Bootstrap starting.
I20260923 20:53:18.622972 1442 tablet_bootstrap.cc:492] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812: Bootstrap starting.
I20260923 20:53:18.623693 1440 log.cc:824] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:18.626834 1440 tablet_bootstrap.cc:492] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca: No bootstrap required, opened a new log
I20260923 20:53:18.626838 1442 tablet_bootstrap.cc:654] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:18.627197 1440 ts_tablet_manager.cc:1397] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:53:18.628470 1441 tablet_bootstrap.cc:654] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:18.628724 1442 log.cc:824] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:18.629968 1441 log.cc:824] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:18.633183 1441 tablet_bootstrap.cc:492] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf: No bootstrap required, opened a new log
I20260923 20:53:18.633498 1441 ts_tablet_manager.cc:1397] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260923 20:53:18.633586 1442 tablet_bootstrap.cc:492] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812: No bootstrap required, opened a new log
I20260923 20:53:18.633935 1442 ts_tablet_manager.cc:1397] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260923 20:53:18.639070 1440 raft_consensus.cc:348] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.639648 1440 raft_consensus.cc:374] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:18.639810 1440 raft_consensus.cc:729] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3049ba088cee41cb9e64f3421cdbc7ca, State: Initialized, Role: FOLLOWER
I20260923 20:53:18.640719 1440 consensus_queue.cc:260] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.642227 1441 raft_consensus.cc:348] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.642704 1441 raft_consensus.cc:374] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:18.642802 1441 raft_consensus.cc:729] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1990a7933d48eda82f1919e8eed1bf, State: Initialized, Role: FOLLOWER
I20260923 20:53:18.642931 1440 ts_tablet_manager.cc:1428] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260923 20:53:18.644064 1441 consensus_queue.cc:260] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.645330 1421 heartbeater.cc:503] Master 127.0.97.62:38221 was elected leader, sending a full tablet report...
I20260923 20:53:18.646135 1441 ts_tablet_manager.cc:1428] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260923 20:53:18.646289 1442 raft_consensus.cc:348] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.646845 1442 raft_consensus.cc:374] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:18.646996 1442 raft_consensus.cc:729] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4564766227384329a9c22b69e577b812, State: Initialized, Role: FOLLOWER
I20260923 20:53:18.647727 1442 consensus_queue.cc:260] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.649158 1287 heartbeater.cc:503] Master 127.0.97.62:38221 was elected leader, sending a full tablet report...
I20260923 20:53:18.650079 1442 ts_tablet_manager.cc:1428] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260923 20:53:18.710249 1154 tablet.cc:2369] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:53:18.730778 1288 tablet.cc:2369] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:53:18.747040 1446 raft_consensus.cc:482] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:18.747395 1446 raft_consensus.cc:504] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.749287 1446 leader_election.cc:288] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc1990a7933d48eda82f1919e8eed1bf (127.0.97.3:39055), 4564766227384329a9c22b69e577b812 (127.0.97.2:45971)
W20260923 20:53:18.754710 1422 tablet.cc:2369] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:53:18.757210 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.749753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43007 (local address 127.0.97.3:39055)
0923 20:53:18.749997 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.750004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.750774 (+ 770us) server_negotiation.cc:407] Connection header received
0923 20:53:18.751036 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.751040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.751103 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.751209 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:18.751880 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.752754 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.755276 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.756233 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.756352 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.756432 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.756874 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.756983 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 20:53:18.757441 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.749720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39055 (local address 127.0.97.1:43007)
0923 20:53:18.750589 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:18.750612 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:18.750835 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:18.751235 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:18.751241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:18.751262 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:18.751711 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.751731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.752908 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.752912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.755081 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.755098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.756458 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.756465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.756648 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.756689 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:53:18.756807 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 20:53:18.756960 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":1,"thread_start_us":479,"threads_started":1}
I20260923 20:53:18.758128 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.749985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51077 (local address 127.0.97.2:45971)
0923 20:53:18.750189 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.750198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.751236 (+ 1038us) server_negotiation.cc:407] Connection header received
0923 20:53:18.751394 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.751399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.751465 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.751534 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:18.752654 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.753408 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.755744 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.757058 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.757210 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.757304 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.757712 (+ 408us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.757883 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 20:53:18.758230 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.749865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45971 (local address 127.0.97.1:51077)
0923 20:53:18.751101 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:18.751116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:18.751254 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:18.751584 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:18.751592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:18.751625 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:18.752465 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.752476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.753560 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.753565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.755570 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:18.755581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.757335 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.757345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:18.757536 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.757575 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:53:18.757682 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:53:18.757807 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":48,"thread_start_us":895,"threads_started":1}
I20260923 20:53:18.758625 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fc1b8aa8d496b861f2fc3e7fc92c0" candidate_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1990a7933d48eda82f1919e8eed1bf" is_pre_election: true
I20260923 20:53:18.759193 1376 raft_consensus.cc:2454] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3049ba088cee41cb9e64f3421cdbc7ca in term 0.
I20260923 20:53:18.759311 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fc1b8aa8d496b861f2fc3e7fc92c0" candidate_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4564766227384329a9c22b69e577b812" is_pre_election: true
I20260923 20:53:18.759843 1242 raft_consensus.cc:2454] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3049ba088cee41cb9e64f3421cdbc7ca in term 0.
I20260923 20:53:18.759931 1042 leader_election.cc:302] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3049ba088cee41cb9e64f3421cdbc7ca, dc1990a7933d48eda82f1919e8eed1bf; no voters:
I20260923 20:53:18.760396 1446 raft_consensus.cc:2781] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:18.760561 1446 raft_consensus.cc:482] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:18.760687 1446 raft_consensus.cc:3037] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:18.763602 1446 raft_consensus.cc:504] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.764355 1446 leader_election.cc:288] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [CANDIDATE]: Term 1 election: Requested vote from peers dc1990a7933d48eda82f1919e8eed1bf (127.0.97.3:39055), 4564766227384329a9c22b69e577b812 (127.0.97.2:45971)
I20260923 20:53:18.764958 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fc1b8aa8d496b861f2fc3e7fc92c0" candidate_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1990a7933d48eda82f1919e8eed1bf"
I20260923 20:53:18.764961 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fc1b8aa8d496b861f2fc3e7fc92c0" candidate_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4564766227384329a9c22b69e577b812"
I20260923 20:53:18.765204 1376 raft_consensus.cc:3037] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:18.765196 1242 raft_consensus.cc:3037] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:18.768579 1376 raft_consensus.cc:2454] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P dc1990a7933d48eda82f1919e8eed1bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3049ba088cee41cb9e64f3421cdbc7ca in term 1.
I20260923 20:53:18.768579 1242 raft_consensus.cc:2454] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 4564766227384329a9c22b69e577b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3049ba088cee41cb9e64f3421cdbc7ca in term 1.
I20260923 20:53:18.769134 1042 leader_election.cc:302] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3049ba088cee41cb9e64f3421cdbc7ca, dc1990a7933d48eda82f1919e8eed1bf; no voters:
I20260923 20:53:18.769508 1446 raft_consensus.cc:2781] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:18.770044 1446 raft_consensus.cc:686] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [term 1 LEADER]: Becoming Leader. State: Replica: 3049ba088cee41cb9e64f3421cdbc7ca, State: Running, Role: LEADER
I20260923 20:53:18.770555 1446 consensus_queue.cc:237] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } }
I20260923 20:53:18.775648 964 catalog_manager.cc:5495] T 1f0fc1b8aa8d496b861f2fc3e7fc92c0 P 3049ba088cee41cb9e64f3421cdbc7ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 3049ba088cee41cb9e64f3421cdbc7ca (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3049ba088cee41cb9e64f3421cdbc7ca" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc1990a7933d48eda82f1919e8eed1bf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4564766227384329a9c22b69e577b812" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45971 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:53:18 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:18.800848 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.783921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.97.1:40879)
0923 20:53:18.784198 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.784207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.784601 (+ 394us) server_negotiation.cc:407] Connection header received
0923 20:53:18.784753 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.784757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.784823 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.784908 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:18.786961 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.787888 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.789312 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.789798 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.789958 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.794384 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:18.794400 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:18.794417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:18.794458 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:18.797560 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.798470 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.798478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.798480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.798716 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.799233 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.799237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.799239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.799612 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:18.799839 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.800231 (+ 392us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.800425 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 20:53:18 dist-test-slave-plcg krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:53:18 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:18.850741 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.834307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.0.97.1:40879)
0923 20:53:18.834579 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.834587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.835062 (+ 475us) server_negotiation.cc:407] Connection header received
0923 20:53:18.835309 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.835314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.835386 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.835495 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:18.837108 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.837894 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.839195 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.839654 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.839796 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.844185 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:18.844192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:18.844198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:18.844246 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:18.847629 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.848428 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.848434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.848438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.848612 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.849051 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.849056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.849058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.849447 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:18.849739 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.850094 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.850342 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
Sep 23 20:53:18 dist-test-slave-plcg krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:53:18 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:18.897781 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.882855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.0.97.1:40879)
0923 20:53:18.883059 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.883068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.883422 (+ 354us) server_negotiation.cc:407] Connection header received
0923 20:53:18.883636 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.883642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.883740 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.883877 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:18.885226 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.886067 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.887281 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.887693 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.887833 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.892042 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:18.892047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:18.892052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:18.892091 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:18.894917 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.895757 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.895762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.895764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.895902 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.896346 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.896350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.896352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.896632 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:18.896827 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.897239 (+ 412us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.897436 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 20:53:18.899668 1068 tablet_service.cc:2908] Rejecting scan request for tablet 1f0fc1b8aa8d496b861f2fc3e7fc92c0: Uninitialized: clean time has not yet been initialized
I20260923 20:53:18.902899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1025
I20260923 20:53:18.916424 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:18.903863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.0.97.1:40879)
0923 20:53:18.904084 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:18.904092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:18.904384 (+ 292us) server_negotiation.cc:407] Connection header received
0923 20:53:18.904594 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:18.904598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:18.904675 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:18.904784 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:18.906046 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.906798 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.908050 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:18.908455 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:18.908592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:18.910975 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:18.910980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:18.910984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:18.911018 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:18.913689 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.914435 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.914442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.914445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.914586 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:18.914978 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:18.914982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:18.914984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:18.915242 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:18.915497 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:18.915849 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 20:53:18.916071 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:53:18.917142 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:19.010740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 1025
I20260923 20:53:19.030241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1159
Sep 23 20:53:19 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:53:19.046043 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:19.031093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.0.97.2:45971)
0923 20:53:19.031253 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:19.031258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:19.031276 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:53:19.031415 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:19.031418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:19.031514 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:19.031607 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:19.032798 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:19.033590 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:19.034984 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:19.035413 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:19.035529 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:19.039706 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:19.039722 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:19.039745 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:19.039787 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:19.043005 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:19.043909 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:19.043917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:19.043919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:19.044063 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:19.044545 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:19.044549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:19.044551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:19.044949 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:19.045181 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:19.045460 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 20:53:19.045636 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:53:19.046698 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:19.135380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 1159
I20260923 20:53:19.152954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1293
Sep 23 20:53:19 dist-test-slave-plcg krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:53:19.169526 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:19.153949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.0.97.3:39055)
0923 20:53:19.154120 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:19.154124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:19.154143 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:53:19.154255 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:19.154259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:19.154321 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:19.154422 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:19.155696 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:19.156457 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:19.158015 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:19.158422 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:19.158742 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:19.163339 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:19.163357 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:19.163375 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:19.163419 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:19.166382 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:19.167193 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:19.167202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:19.167204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:19.167375 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:19.167822 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:19.167826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:19.167828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:19.168214 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:19.168454 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:19.168825 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 20:53:19.169019 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:53:19.170285 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:19.259050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 1293
I20260923 20:53:19.275498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 932
I20260923 20:53:19.276801 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:19.384934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 932
2026-09-23T20:53:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4278 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:19 dist-test-slave-plcg krb5kdc[1470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:19 dist-test-slave-plcg krb5kdc[1470](info): set up 2 sockets
Sep 23 20:53:19 dist-test-slave-plcg krb5kdc[1470](info): commencing operation
krb5kdc: starting...
W20260923 20:53:20.473940 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:20 dist-test-slave-plcg krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:20.666682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34519
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:21.009908 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:21.010265 1483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:21.010442 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:21.018352 1483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:21.018482 1483 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:53:21.018563 1483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:21.018684 1483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:21.018792 1483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:21.018906 1483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:21.031186 1483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34519
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:21.032627 1483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:21.034739 1483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:21.049638 1491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:21.049710 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:21.049593 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:21.051250 1483 server_base.cc:1034] running on GCE node
I20260923 20:53:21.052196 1483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:21.053915 1483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:21.055182 1483 hybrid_clock.cc:648] HybridClock initialized: now 1790196801055102 us; error 63 us; skew 500 ppm
I20260923 20:53:21.055624 1483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:21 dist-test-slave-plcg krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:21.059468 1483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:21.061117 1483 webserver.cc:460] Webserver started at http://127.0.97.62:40735/ using document root <none> and password file <none>
I20260923 20:53:21.062000 1483 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:21.062106 1483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:21.062481 1483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:21.065739 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "b74ad04dc444439587d6d12c15919825"
format_stamp: "Formatted at 2026-09-23 20:53:21 on dist-test-slave-plcg"
server_key: "ec17ff11cbd4b8d3a529b716ba92f495"
server_key_iv: "991a89e6acd35fce47b72445d032ef0a"
server_key_version: "encryptionkey@0"
I20260923 20:53:21.066772 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "b74ad04dc444439587d6d12c15919825"
format_stamp: "Formatted at 2026-09-23 20:53:21 on dist-test-slave-plcg"
server_key: "ec17ff11cbd4b8d3a529b716ba92f495"
server_key_iv: "991a89e6acd35fce47b72445d032ef0a"
server_key_version: "encryptionkey@0"
I20260923 20:53:21.072613 1483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:21.077270 1498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:21.079093 1483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:53:21.079339 1483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "b74ad04dc444439587d6d12c15919825"
format_stamp: "Formatted at 2026-09-23 20:53:21 on dist-test-slave-plcg"
server_key: "ec17ff11cbd4b8d3a529b716ba92f495"
server_key_iv: "991a89e6acd35fce47b72445d032ef0a"
server_key_version: "encryptionkey@0"
I20260923 20:53:21.079602 1483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:21.114065 1483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:21.118881 1483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:21.119122 1483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:21.141743 1483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34519
I20260923 20:53:21.141760 1559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34519 every 8 connection(s)
I20260923 20:53:21.143226 1483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:21.146445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 1483
I20260923 20:53:21.146690 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:21.147289 1560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:21.147375 388 external_mini_cluster.cc:1428] Setting key c63dd53be1fe92f98f039d3c90b8debf
I20260923 20:53:21.162285 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825: Bootstrap starting.
Sep 23 20:53:21 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:21.165807 1560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:21.167069 1560 log.cc:824] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:21.170653 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825: No bootstrap required, opened a new log
I20260923 20:53:21.172483 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:21.149530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.0.97.62:34519)
0923 20:53:21.150488 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:21.150501 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:21.150550 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:53:21.152363 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:21.152394 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:21.152864 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:21.153290 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:21.155201 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.156605 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.158429 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.159205 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.159366 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:21.164266 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:21.164277 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:21.164293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:21.164339 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:21.167634 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:21.168519 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:21.168529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:21.168538 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:21.168757 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:21.169140 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:21.169144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:21.169147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:21.169900 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:21.170213 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:21.170567 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 20:53:21.171043 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":503,"threads_started":1}
I20260923 20:53:21.180068 1560 raft_consensus.cc:348] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74ad04dc444439587d6d12c15919825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260923 20:53:21.180447 1560 raft_consensus.cc:374] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:21.180521 1560 raft_consensus.cc:729] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b74ad04dc444439587d6d12c15919825, State: Initialized, Role: FOLLOWER
I20260923 20:53:21.181113 1560 consensus_queue.cc:260] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74ad04dc444439587d6d12c15919825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260923 20:53:21.181329 1560 raft_consensus.cc:388] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:21.181432 1560 raft_consensus.cc:482] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:21.181559 1560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:21.184156 1560 raft_consensus.cc:504] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74ad04dc444439587d6d12c15919825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260923 20:53:21.184722 1560 leader_election.cc:302] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b74ad04dc444439587d6d12c15919825; no voters:
I20260923 20:53:21.185292 1560 leader_election.cc:288] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:21.185444 1565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:21.186218 1565 raft_consensus.cc:686] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [term 1 LEADER]: Becoming Leader. State: Replica: b74ad04dc444439587d6d12c15919825, State: Running, Role: LEADER
I20260923 20:53:21.186796 1565 consensus_queue.cc:237] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74ad04dc444439587d6d12c15919825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } }
I20260923 20:53:21.187534 1560 sys_catalog.cc:564] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:21.189985 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b74ad04dc444439587d6d12c15919825. Latest consensus state: current_term: 1 leader_uuid: "b74ad04dc444439587d6d12c15919825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74ad04dc444439587d6d12c15919825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } } }
I20260923 20:53:21.190471 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:21.190007 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b74ad04dc444439587d6d12c15919825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74ad04dc444439587d6d12c15919825" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34519 } } }
I20260923 20:53:21.190721 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:21.194583 1574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:21.200107 1574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:21.209672 1574 catalog_manager.cc:1347] Generated new cluster ID: c286a327f7c446039011183a1a2c3d8d
I20260923 20:53:21.209843 1574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:21.224254 1574 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:21.225651 1574 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:21.234280 1574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825: Generated new TSK 0
I20260923 20:53:21.235421 1574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:21.294379 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34519
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:21.635078 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:21.635443 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:21.635622 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:21.643144 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:21.643326 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:21.643488 1586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:53:21.643630 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:21.654032 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:21.655390 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:21.656847 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:21.670830 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:21.670889 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:21.670801 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:21.671957 1586 server_base.cc:1034] running on GCE node
I20260923 20:53:21.672623 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:21.673791 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:21.675117 1586 hybrid_clock.cc:648] HybridClock initialized: now 1790196801675037 us; error 58 us; skew 500 ppm
I20260923 20:53:21.675472 1586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:21 dist-test-slave-plcg krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:21.678957 1586 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:21.680567 1586 webserver.cc:460] Webserver started at http://127.0.97.1:37721/ using document root <none> and password file <none>
I20260923 20:53:21.681334 1586 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:21.681471 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:21.681886 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:21.685073 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "bbc33d677eaf4fda8ff7e1db22985940"
format_stamp: "Formatted at 2026-09-23 20:53:21 on dist-test-slave-plcg"
server_key: "b7a5de189abcd55cb7e353269809964d"
server_key_iv: "7b0734eca9f9992bfad7e404630cd66e"
server_key_version: "encryptionkey@0"
I20260923 20:53:21.686026 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bbc33d677eaf4fda8ff7e1db22985940"
format_stamp: "Formatted at 2026-09-23 20:53:21 on dist-test-slave-plcg"
server_key: "b7a5de189abcd55cb7e353269809964d"
server_key_iv: "7b0734eca9f9992bfad7e404630cd66e"
server_key_version: "encryptionkey@0"
I20260923 20:53:21.692010 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260923 20:53:21.695856 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:21.697458 1586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:53:21.697678 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "bbc33d677eaf4fda8ff7e1db22985940"
format_stamp: "Formatted at 2026-09-23 20:53:21 on dist-test-slave-plcg"
server_key: "b7a5de189abcd55cb7e353269809964d"
server_key_iv: "7b0734eca9f9992bfad7e404630cd66e"
server_key_version: "encryptionkey@0"
I20260923 20:53:21.697999 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:21.715973 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:21.720922 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:21.721179 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:21.724053 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:21.724197 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:21.724339 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:21.724409 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:53:21 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:21.750613 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:21.724887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33619 (local address 127.0.97.62:34519)
0923 20:53:21.725438 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:21.725442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:21.726923 (+ 1481us) server_negotiation.cc:407] Connection header received
0923 20:53:21.729627 (+ 2704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:21.729633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:21.729736 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:21.729886 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:21.732546 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.733575 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.736315 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.736798 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.736944 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:21.743112 (+ 6168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:21.743118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:21.743122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:21.743160 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:21.746495 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:21.747351 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:21.747359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:21.747365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:21.747537 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:21.748193 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:21.748202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:21.748205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:21.748666 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:21.748872 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:21.749992 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0923 20:53:21.750241 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":334,"threads_started":1}
I20260923 20:53:21.751721 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:21.725775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.97.1:33619)
0923 20:53:21.726743 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:21.726786 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:21.729187 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:21.730220 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:21.730235 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:21.731250 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:21.732297 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:21.732339 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.733847 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.733856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:21.736112 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:21.736128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.737166 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.737173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:21.737561 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:21.738613 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:21.738638 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:21.742649 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:21.746679 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:21.746691 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:21.746711 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:21.747197 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:21.747704 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:21.747709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:21.747713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:21.748020 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:21.748849 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:21.748861 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:21.749373 (+ 512us) client_negotiation.cc:769] Sending connection context
0923 20:53:21.749888 (+ 515us) client_negotiation.cc:240] Negotiation successful
0923 20:53:21.750492 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":400,"threads_started":1}
I20260923 20:53:21.778483 1586 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45505
I20260923 20:53:21.778483 1721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45505 every 8 connection(s)
I20260923 20:53:21.779987 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:21.780643 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:21.759128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39063 (local address 127.0.97.62:34519)
0923 20:53:21.759322 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:21.759326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:21.759946 (+ 620us) server_negotiation.cc:407] Connection header received
0923 20:53:21.760191 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:21.760195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:21.760284 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:21.760404 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:21.762194 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.762892 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.765558 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.766045 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.766207 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:21.768832 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:21.768837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:21.768840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:21.768874 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:21.777488 (+ 8614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:21.778494 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:21.778502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:21.778506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:21.778659 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:21.779047 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:21.779053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:21.779056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:21.779346 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:21.779509 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:21.780029 (+ 520us) server_negotiation.cc:299] Negotiation successful
0923 20:53:21.780276 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:53:21.780783 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:21.759197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.97.1:39063)
0923 20:53:21.759833 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:21.759850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:21.760029 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:21.760476 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:21.760483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:21.761392 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:21.761971 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:21.761991 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.763070 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.763079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:21.765346 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:21.765371 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:21.766401 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:21.766419 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:21.766545 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:21.767461 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:21.767487 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:21.768603 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:21.777691 (+ 9088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:21.777699 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:21.777705 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:21.778342 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:21.778786 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:21.778792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:21.778796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:21.778929 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:21.779475 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:21.779479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:21.779749 (+ 270us) client_negotiation.cc:769] Sending connection context
0923 20:53:21.779909 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 20:53:21.780279 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
I20260923 20:53:21.782908 1722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34519
I20260923 20:53:21.783201 1722 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:21.784011 1722 heartbeater.cc:511] Master 127.0.97.62:34519 requested a full tablet report, sending...
I20260923 20:53:21.785260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 1586
I20260923 20:53:21.785476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:21.786195 388 external_mini_cluster.cc:1428] Setting key 9d8ff432b096ff769dc9790cb223bc67
I20260923 20:53:21.786550 1514 ts_manager.cc:194] Registered new tserver with Master: bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1:45505)
I20260923 20:53:21.788411 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33619
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:21.829571 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34519
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:22.171466 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:22.171816 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:22.171993 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:22.179494 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:22.179667 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:22.179829 1726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:53:22.179971 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:22.190516 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:22.191915 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:22.193370 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:22.207494 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:22.207437 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:22.208051 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:22.208724 1726 server_base.cc:1034] running on GCE node
I20260923 20:53:22.209336 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:22.210345 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:22.211577 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790196802211510 us; error 37 us; skew 500 ppm
I20260923 20:53:22.211861 1726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:22 dist-test-slave-plcg krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:22.215265 1726 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:22.216856 1726 webserver.cc:460] Webserver started at http://127.0.97.2:33697/ using document root <none> and password file <none>
I20260923 20:53:22.217571 1726 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:22.217687 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:22.218052 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:22.221014 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "b324980155884832b93867f84c33b62b"
format_stamp: "Formatted at 2026-09-23 20:53:22 on dist-test-slave-plcg"
server_key: "47ebdfc4a3c7233be2ce89bf0396b83b"
server_key_iv: "73238fc9bb287312247569ee0a3e1a6d"
server_key_version: "encryptionkey@0"
I20260923 20:53:22.221872 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b324980155884832b93867f84c33b62b"
format_stamp: "Formatted at 2026-09-23 20:53:22 on dist-test-slave-plcg"
server_key: "47ebdfc4a3c7233be2ce89bf0396b83b"
server_key_iv: "73238fc9bb287312247569ee0a3e1a6d"
server_key_version: "encryptionkey@0"
I20260923 20:53:22.227578 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:53:22.231354 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:22.232939 1726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260923 20:53:22.233136 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "b324980155884832b93867f84c33b62b"
format_stamp: "Formatted at 2026-09-23 20:53:22 on dist-test-slave-plcg"
server_key: "47ebdfc4a3c7233be2ce89bf0396b83b"
server_key_iv: "73238fc9bb287312247569ee0a3e1a6d"
server_key_version: "encryptionkey@0"
I20260923 20:53:22.233398 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:22.277101 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:22.281502 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:22.281865 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:22.284858 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:22.284978 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:22.285118 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:22.285184 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:53:22 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:22.310902 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.285846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38447 (local address 127.0.97.62:34519)
0923 20:53:22.286638 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:22.286643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:22.287684 (+ 1041us) server_negotiation.cc:407] Connection header received
0923 20:53:22.290055 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:22.290061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:22.290162 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:22.290283 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:22.292813 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.294239 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.296055 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.296863 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.297023 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.302903 (+ 5880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:22.302909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:22.302912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:22.302948 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:22.306917 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.308122 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.308129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.308133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.308342 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.308830 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.308836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.308840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.309134 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:22.309318 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:22.310226 (+ 908us) server_negotiation.cc:299] Negotiation successful
0923 20:53:22.310482 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":575,"threads_started":1}
I20260923 20:53:22.312055 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.286574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.97.2:38447)
0923 20:53:22.287509 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:22.287551 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:22.289673 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:22.290569 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:22.290587 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:22.291508 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:22.292567 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.292604 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.294450 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.294456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.295870 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.295885 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.297184 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.297190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.297394 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.298299 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:22.298345 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:22.302622 (+ 4277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:22.307168 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.307181 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.307201 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.307967 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.308474 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.308480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.308483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.308707 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.309292 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:22.309301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:22.309696 (+ 395us) client_negotiation.cc:769] Sending connection context
0923 20:53:22.310148 (+ 452us) client_negotiation.cc:240] Negotiation successful
0923 20:53:22.310750 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":395,"threads_started":1}
I20260923 20:53:22.335083 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.318775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.97.2:53965)
0923 20:53:22.319507 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:22.319536 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:22.319722 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:22.320185 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:22.320191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:22.320805 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:22.321107 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.321118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.322827 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.322834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.324535 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.324554 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.325678 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.325686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.325914 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.326736 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:22.326772 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:22.328001 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:22.331644 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.331654 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.331659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.332343 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.332858 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.332863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.332866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.333077 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.333941 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:22.333946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:22.334143 (+ 197us) client_negotiation.cc:769] Sending connection context
0923 20:53:22.334334 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 20:53:22.334722 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":381,"threads_started":1}
I20260923 20:53:22.336036 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.318777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53965 (local address 127.0.97.62:34519)
0923 20:53:22.319017 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:22.319026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:22.319636 (+ 610us) server_negotiation.cc:407] Connection header received
0923 20:53:22.319922 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:22.319927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:22.320012 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:22.320141 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:22.321289 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.322627 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.324807 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.325437 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.325611 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.328221 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:22.328230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:22.328239 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:22.328273 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:22.331467 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.332558 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.332566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.332571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.332728 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.333250 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.333255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.333258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.333679 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:22.333873 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:22.335444 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0923 20:53:22.335737 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:53:22.336820 1726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46753
I20260923 20:53:22.336839 1861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46753 every 8 connection(s)
I20260923 20:53:22.338351 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:22.339073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 1726
I20260923 20:53:22.339277 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:22.339834 388 external_mini_cluster.cc:1428] Setting key 6dc1f5ee89ed0911c8e4a39529bc9211
I20260923 20:53:22.341233 1862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34519
I20260923 20:53:22.341543 1862 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:22.342845 1862 heartbeater.cc:511] Master 127.0.97.62:34519 requested a full tablet report, sending...
I20260923 20:53:22.344834 1514 ts_manager.cc:194] Registered new tserver with Master: b324980155884832b93867f84c33b62b (127.0.97.2:46753)
I20260923 20:53:22.346131 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38447
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:22.381675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34519
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:53:22.792421 1722 heartbeater.cc:503] Master 127.0.97.62:34519 was elected leader, sending a full tablet report...
W20260923 20:53:22.794540 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:22.794981 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:22.795192 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:22.811720 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:22.811916 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:22.812111 1866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:53:22.812283 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:22.833979 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:22.835861 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:22.837936 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:22.855343 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:22.855370 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:22.856267 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:22.856799 1866 server_base.cc:1034] running on GCE node
I20260923 20:53:22.857527 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:22.858644 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:22.859948 1866 hybrid_clock.cc:648] HybridClock initialized: now 1790196802859861 us; error 77 us; skew 500 ppm
I20260923 20:53:22.860285 1866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:22 dist-test-slave-plcg krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:22.864147 1866 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:22.866348 1866 webserver.cc:460] Webserver started at http://127.0.97.3:34491/ using document root <none> and password file <none>
I20260923 20:53:22.867453 1866 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:22.867594 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:22.868019 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:22.873010 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "1f6c34254cee41958872e26c24cbfbcd"
format_stamp: "Formatted at 2026-09-23 20:53:22 on dist-test-slave-plcg"
server_key: "e05b7f2fe5135cccfd99f2697d0ab7f6"
server_key_iv: "65d6cfc9e6640a8867a0eee6c9542ea5"
server_key_version: "encryptionkey@0"
I20260923 20:53:22.874342 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1f6c34254cee41958872e26c24cbfbcd"
format_stamp: "Formatted at 2026-09-23 20:53:22 on dist-test-slave-plcg"
server_key: "e05b7f2fe5135cccfd99f2697d0ab7f6"
server_key_iv: "65d6cfc9e6640a8867a0eee6c9542ea5"
server_key_version: "encryptionkey@0"
I20260923 20:53:22.882635 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260923 20:53:22.888422 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:22.890669 1866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:53:22.890915 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "1f6c34254cee41958872e26c24cbfbcd"
format_stamp: "Formatted at 2026-09-23 20:53:22 on dist-test-slave-plcg"
server_key: "e05b7f2fe5135cccfd99f2697d0ab7f6"
server_key_iv: "65d6cfc9e6640a8867a0eee6c9542ea5"
server_key_version: "encryptionkey@0"
I20260923 20:53:22.891239 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:22.918107 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:22.924619 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:22.924984 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:22.929041 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:22.929220 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:53:22.929430 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:22.929549 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:53:22 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:22.969295 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.929853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.97.3:47615)
0923 20:53:22.941368 (+ 11515us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:22.941441 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:22.943906 (+ 2465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:22.944782 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:22.944795 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:22.945404 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:22.946398 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.946434 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.948148 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.948156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.951251 (+ 3095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.951280 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.952996 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.953005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.953331 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.954392 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:22.954422 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:22.958816 (+ 4394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:22.963190 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.963201 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.963225 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.964075 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.964659 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.964666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.964676 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.964969 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.965652 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:22.965660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:22.966138 (+ 478us) client_negotiation.cc:769] Sending connection context
0923 20:53:22.966552 (+ 414us) client_negotiation.cc:240] Negotiation successful
0923 20:53:22.967190 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11089,"thread_start_us":11927,"threads_started":1}
I20260923 20:53:22.970242 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.929010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47615 (local address 127.0.97.62:34519)
0923 20:53:22.941394 (+ 12384us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:22.941401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:22.941901 (+ 500us) server_negotiation.cc:407] Connection header received
0923 20:53:22.944248 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:22.944253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:22.944358 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:22.944487 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:22.946730 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.947906 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.951533 (+ 3627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.952651 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.952871 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.959245 (+ 6374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:22.959252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:22.959259 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:22.959314 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:22.962963 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.964295 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.964301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.964305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.964457 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.965159 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.965167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.965173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.965516 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:22.965688 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:22.969464 (+ 3776us) server_negotiation.cc:299] Negotiation successful
0923 20:53:22.969751 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12260,"thread_start_us":12391,"threads_started":1}
I20260923 20:53:22.998679 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.978131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57069 (local address 127.0.97.62:34519)
0923 20:53:22.978521 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:22.978530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:22.982717 (+ 4187us) server_negotiation.cc:407] Connection header received
0923 20:53:22.982934 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:22.982938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:22.983011 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:22.983172 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:22.984770 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.986134 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.987600 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.988539 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.988732 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.991172 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:22.991181 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:22.991187 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:22.991239 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:22.994641 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.995657 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.995666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.995671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.995871 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:22.996528 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:22.996535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:22.996540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:22.996997 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:22.997183 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:22.998155 (+ 972us) server_negotiation.cc:299] Negotiation successful
0923 20:53:22.998412 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260923 20:53:22.999226 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:22.978147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.97.3:57069)
0923 20:53:22.982522 (+ 4375us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:22.982553 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:22.982791 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:22.983206 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:22.983211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:22.984067 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:22.984564 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.984581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.986336 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.986342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.987423 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:22.987437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:22.988767 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:22.988771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:22.988889 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:22.989657 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:22.989677 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:22.990902 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:22.994815 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.994821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.994825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.995484 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.996059 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:22.996085 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:22.996092 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:22.996313 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:22.997387 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:22.997397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:22.997697 (+ 300us) client_negotiation.cc:769] Sending connection context
0923 20:53:22.997991 (+ 294us) client_negotiation.cc:240] Negotiation successful
0923 20:53:22.998512 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4144,"thread_start_us":3886,"threads_started":1}
I20260923 20:53:23.004556 2001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38661 every 8 connection(s)
I20260923 20:53:23.004551 1866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38661
I20260923 20:53:23.005995 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:23.008285 2002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34519
I20260923 20:53:23.008679 2002 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:23.009477 2002 heartbeater.cc:511] Master 127.0.97.62:34519 requested a full tablet report, sending...
I20260923 20:53:23.011133 1514 ts_manager.cc:194] Registered new tserver with Master: 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3:38661)
I20260923 20:53:23.011948 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47615
I20260923 20:53:23.014550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 1866
I20260923 20:53:23.014720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:23.015244 388 external_mini_cluster.cc:1428] Setting key ca715505cf3976e6d7b3d84357209ddc
I20260923 20:53:23.018430 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:53:23 dist-test-slave-plcg krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:53:23 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:23.067152 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.052454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.0.97.62:34519)
0923 20:53:23.052696 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.052702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.052918 (+ 216us) server_negotiation.cc:407] Connection header received
0923 20:53:23.053145 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.053156 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.053219 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.053332 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:23.055040 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.055811 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.057149 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.057541 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.057688 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.061645 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:23.061650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:23.061654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:23.061707 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:23.064342 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:23.065052 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:23.065064 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:23.065067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:23.065194 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:23.065600 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:23.065606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:23.065609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:23.065979 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:23.066245 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.066551 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.066785 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:53:23.071607 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:23.078989 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:23.113766 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.092908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.0.97.1:45505)
0923 20:53:23.106110 (+ 13202us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.106119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.106154 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:53:23.106268 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.106279 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.106705 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.106957 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.107688 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.109551 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.111556 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.112513 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.112696 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.112928 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.113229 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.113418 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12996,"thread_start_us":12723,"threads_started":1}
I20260923 20:53:23.113859 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.092774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45505 (local address 127.0.0.1:54078)
0923 20:53:23.093510 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.093562 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.093737 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.107041 (+ 13304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.107047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.107076 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.107497 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.107509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.109730 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.109735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.111379 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.111392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.112740 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.112746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.112953 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.113009 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.113181 (+ 172us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.113418 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":1,"thread_start_us":316,"threads_started":1}
I20260923 20:53:23.114454 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.092850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38661 (local address 127.0.0.1:58314)
0923 20:53:23.105164 (+ 12314us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.105190 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.105395 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.106257 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.106268 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.106307 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.106648 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.106659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.108454 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.108459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.110291 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.110310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.113154 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.113163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.113523 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.113582 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.113688 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.113920 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12119,"mutex_wait_us":61,"thread_start_us":11945,"threads_started":1}
I20260923 20:53:23.115425 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.094229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.0.97.3:38661)
0923 20:53:23.104818 (+ 10589us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.104829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.105356 (+ 527us) server_negotiation.cc:407] Connection header received
0923 20:53:23.105620 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.105631 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.105953 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.106217 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.106848 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.108303 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.110500 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.112505 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.112673 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.112970 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.114756 (+ 1786us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.114988 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10325,"thread_start_us":10086,"threads_started":1}
I20260923 20:53:23.119648 1932 tablet_service.cc:1467] Processing CreateTablet for tablet dc0136c5466b46b1b33740ac892814b6 (DEFAULT_TABLE table=test-table [id=fd5a65f66fea43718e2dd290b3b21f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:23.121250 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0136c5466b46b1b33740ac892814b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.123679 1652 tablet_service.cc:1467] Processing CreateTablet for tablet dc0136c5466b46b1b33740ac892814b6 (DEFAULT_TABLE table=test-table [id=fd5a65f66fea43718e2dd290b3b21f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:23.125152 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0136c5466b46b1b33740ac892814b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.125314 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.095690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46753 (local address 127.0.0.1:55950)
0923 20:53:23.114194 (+ 18504us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.114215 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.114366 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.119029 (+ 4663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.119038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.119089 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.119475 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.119489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.121250 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.121257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.122818 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.122832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.124588 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.124593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.124722 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.124747 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.124857 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.125010 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18355,"thread_start_us":23663,"threads_started":1}
I20260923 20:53:23.125365 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.096290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.0.97.2:46753)
0923 20:53:23.118277 (+ 21987us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.118286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.118321 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:53:23.118466 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.118474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.118748 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.118986 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.119676 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.121130 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.122988 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.124123 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.124254 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.124504 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.124925 (+ 421us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.125101 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21825,"thread_start_us":21705,"threads_started":1}
I20260923 20:53:23.129398 1792 tablet_service.cc:1467] Processing CreateTablet for tablet dc0136c5466b46b1b33740ac892814b6 (DEFAULT_TABLE table=test-table [id=fd5a65f66fea43718e2dd290b3b21f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:23.131037 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0136c5466b46b1b33740ac892814b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.137275 2021 tablet_bootstrap.cc:492] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Bootstrap starting.
I20260923 20:53:23.139410 2022 tablet_bootstrap.cc:492] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Bootstrap starting.
I20260923 20:53:23.142546 2021 tablet_bootstrap.cc:654] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.143105 2022 tablet_bootstrap.cc:654] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.143894 2021 log.cc:824] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:23.144389 2022 log.cc:824] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:23.153594 2021 tablet_bootstrap.cc:492] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: No bootstrap required, opened a new log
I20260923 20:53:23.154173 2021 ts_tablet_manager.cc:1397] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20260923 20:53:23.154690 2022 tablet_bootstrap.cc:492] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: No bootstrap required, opened a new log
I20260923 20:53:23.155160 2022 ts_tablet_manager.cc:1397] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260923 20:53:23.161301 2025 tablet_bootstrap.cc:492] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Bootstrap starting.
I20260923 20:53:23.164731 2025 tablet_bootstrap.cc:654] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.166241 2025 log.cc:824] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:23.167042 2021 raft_consensus.cc:348] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.167665 2021 raft_consensus.cc:374] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.167840 2021 raft_consensus.cc:729] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6c34254cee41958872e26c24cbfbcd, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.168699 2021 consensus_queue.cc:260] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.169968 2002 heartbeater.cc:503] Master 127.0.97.62:34519 was elected leader, sending a full tablet report...
I20260923 20:53:23.170753 2021 ts_tablet_manager.cc:1428] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
I20260923 20:53:23.170958 2025 tablet_bootstrap.cc:492] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: No bootstrap required, opened a new log
I20260923 20:53:23.171265 2025 ts_tablet_manager.cc:1397] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:53:23.174121 2022 raft_consensus.cc:348] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.174665 2022 raft_consensus.cc:374] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.174798 2022 raft_consensus.cc:729] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc33d677eaf4fda8ff7e1db22985940, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.175532 2022 consensus_queue.cc:260] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.177547 2022 ts_tablet_manager.cc:1428] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Time spent starting tablet: real 0.022s user 0.011s sys 0.004s
I20260923 20:53:23.180728 2028 raft_consensus.cc:482] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:23.181047 2028 raft_consensus.cc:504] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.182992 2028 leader_election.cc:288] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3:38661), b324980155884832b93867f84c33b62b (127.0.97.2:46753)
I20260923 20:53:23.183246 2025 raft_consensus.cc:348] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.184160 2025 raft_consensus.cc:374] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.184389 2025 raft_consensus.cc:729] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b324980155884832b93867f84c33b62b, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.185225 2025 consensus_queue.cc:260] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.186568 1862 heartbeater.cc:503] Master 127.0.97.62:34519 was elected leader, sending a full tablet report...
I20260923 20:53:23.187429 2025 ts_tablet_manager.cc:1428] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260923 20:53:23.192198 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.183255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46753 (local address 127.0.97.1:56875)
0923 20:53:23.184121 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.184141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.184374 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.185054 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.185059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.185151 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.185512 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.185530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.187077 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.187082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.189488 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.189503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.191274 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.191281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.191466 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.191499 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.191610 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.191769 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":54,"spinlock_wait_cycles":26496,"thread_start_us":835,"threads_started":1}
I20260923 20:53:23.192225 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.183440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56875 (local address 127.0.97.2:46753)
0923 20:53:23.183636 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.183642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.184227 (+ 585us) server_negotiation.cc:407] Connection header received
0923 20:53:23.184548 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.184556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.184638 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.184730 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.185684 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.186896 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.189678 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.191023 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.191171 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.191281 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.191861 (+ 580us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.191998 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 20:53:23.192335 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.183959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41787 (local address 127.0.97.3:38661)
0923 20:53:23.184135 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.184142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.185087 (+ 945us) server_negotiation.cc:407] Connection header received
0923 20:53:23.185213 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.185219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.185299 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.185402 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.186115 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.187161 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.190065 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.191349 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.191508 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.191628 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.191988 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.192123 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 20:53:23.193063 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.183360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38661 (local address 127.0.97.1:41787)
0923 20:53:23.184972 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.184989 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.185100 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.185438 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.185443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.185473 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.185911 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.185925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.187361 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.187368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.189890 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.189905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.191579 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.191586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.191762 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.191835 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.192593 (+ 758us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.192730 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"thread_start_us":1323,"threads_started":1}
I20260923 20:53:23.193593 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0136c5466b46b1b33740ac892814b6" candidate_uuid: "bbc33d677eaf4fda8ff7e1db22985940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b324980155884832b93867f84c33b62b" is_pre_election: true
I20260923 20:53:23.194116 1817 raft_consensus.cc:2454] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc33d677eaf4fda8ff7e1db22985940 in term 0.
I20260923 20:53:23.194093 1956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0136c5466b46b1b33740ac892814b6" candidate_uuid: "bbc33d677eaf4fda8ff7e1db22985940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6c34254cee41958872e26c24cbfbcd" is_pre_election: true
I20260923 20:53:23.194630 1956 raft_consensus.cc:2454] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc33d677eaf4fda8ff7e1db22985940 in term 0.
I20260923 20:53:23.194820 1603 leader_election.cc:302] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b324980155884832b93867f84c33b62b, bbc33d677eaf4fda8ff7e1db22985940; no voters:
I20260923 20:53:23.195525 2028 raft_consensus.cc:2781] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:23.195677 2028 raft_consensus.cc:482] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:23.195796 2028 raft_consensus.cc:3037] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.198185 2028 raft_consensus.cc:504] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.198868 2028 leader_election.cc:288] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [CANDIDATE]: Term 1 election: Requested vote from peers 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3:38661), b324980155884832b93867f84c33b62b (127.0.97.2:46753)
I20260923 20:53:23.199430 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0136c5466b46b1b33740ac892814b6" candidate_uuid: "bbc33d677eaf4fda8ff7e1db22985940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b324980155884832b93867f84c33b62b"
I20260923 20:53:23.199571 1956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0136c5466b46b1b33740ac892814b6" candidate_uuid: "bbc33d677eaf4fda8ff7e1db22985940" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6c34254cee41958872e26c24cbfbcd"
I20260923 20:53:23.199658 1817 raft_consensus.cc:3037] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.199851 1956 raft_consensus.cc:3037] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.202200 1817 raft_consensus.cc:2454] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc33d677eaf4fda8ff7e1db22985940 in term 1.
I20260923 20:53:23.202592 1956 raft_consensus.cc:2454] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc33d677eaf4fda8ff7e1db22985940 in term 1.
I20260923 20:53:23.202739 1603 leader_election.cc:302] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b324980155884832b93867f84c33b62b, bbc33d677eaf4fda8ff7e1db22985940; no voters:
I20260923 20:53:23.203179 2028 raft_consensus.cc:2781] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:23.203625 2028 raft_consensus.cc:686] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 LEADER]: Becoming Leader. State: Replica: bbc33d677eaf4fda8ff7e1db22985940, State: Running, Role: LEADER
I20260923 20:53:23.204125 2028 consensus_queue.cc:237] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } }
I20260923 20:53:23.208922 1514 catalog_manager.cc:5495] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bbc33d677eaf4fda8ff7e1db22985940" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:23.249650 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.243774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.0.97.1:45505)
0923 20:53:23.243984 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.243990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.244012 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:53:23.244451 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.244457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.244602 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.244775 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:23.245274 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.246091 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.247398 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.247801 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.247895 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.248367 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.248886 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.249102 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.249253 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.249356 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:53:23.257686 2003 tablet.cc:2369] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:53:23.258816 1956 raft_consensus.cc:1261] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Refusing update from remote peer bbc33d677eaf4fda8ff7e1db22985940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:23.258816 1817 raft_consensus.cc:1261] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Refusing update from remote peer bbc33d677eaf4fda8ff7e1db22985940: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:23.259886 2028 consensus_queue.cc:1035] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:23.260237 2032 consensus_queue.cc:1035] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:23.274596 2036 mvcc.cc:205] Tried to move back new op lower bound from 7332646106135326720 to 7332646105927888896. Current Snapshot: MvccSnapshot[applied={T|T < 7332646106135326720}]
I20260923 20:53:23.295787 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:23.296089 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:23.300172 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825: Sending DeleteTablet for 3 replicas of tablet dc0136c5466b46b1b33740ac892814b6
I20260923 20:53:23.301379 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0136c5466b46b1b33740ac892814b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54078
I20260923 20:53:23.301584 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0136c5466b46b1b33740ac892814b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58314
I20260923 20:53:23.301892 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0136c5466b46b1b33740ac892814b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55950
I20260923 20:53:23.303221 1515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:23.304055 1515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:23.311045 2047 tablet_replica.cc:330] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: stopping tablet replica
I20260923 20:53:23.311327 2046 tablet_replica.cc:330] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: stopping tablet replica
I20260923 20:53:23.311842 2046 raft_consensus.cc:2229] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:23.312353 2046 raft_consensus.cc:2258] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:23.312369 2047 raft_consensus.cc:2229] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:23.313468 2047 raft_consensus.cc:2258] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:23.313654 2045 tablet_replica.cc:330] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: stopping tablet replica
I20260923 20:53:23.314097 2045 raft_consensus.cc:2229] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:23.315029 1652 tablet_service.cc:1467] Processing CreateTablet for tablet bfb722776df649998ea4cde70c248585 (DEFAULT_TABLE table=test-table [id=775744b02f414a1db54e2e686db26783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:23.315199 1792 tablet_service.cc:1467] Processing CreateTablet for tablet bfb722776df649998ea4cde70c248585 (DEFAULT_TABLE table=test-table [id=775744b02f414a1db54e2e686db26783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:23.315351 2046 ts_tablet_manager.cc:1905] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:23.315641 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb722776df649998ea4cde70c248585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.315685 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb722776df649998ea4cde70c248585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.322533 1932 tablet_service.cc:1467] Processing CreateTablet for tablet bfb722776df649998ea4cde70c248585 (DEFAULT_TABLE table=test-table [id=775744b02f414a1db54e2e686db26783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:23.322737 2046 ts_tablet_manager.cc:1918] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:23.322932 2046 log.cc:1196] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/dc0136c5466b46b1b33740ac892814b6
I20260923 20:53:23.323092 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb722776df649998ea4cde70c248585. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.323570 2046 ts_tablet_manager.cc:1939] T dc0136c5466b46b1b33740ac892814b6 P 1f6c34254cee41958872e26c24cbfbcd: Deleting consensus metadata
I20260923 20:53:23.324741 2047 ts_tablet_manager.cc:1905] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:23.325119 2022 tablet_bootstrap.cc:492] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: Bootstrap starting.
I20260923 20:53:23.326262 2045 raft_consensus.cc:2258] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:23.327047 2022 tablet_bootstrap.cc:654] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.328466 2045 ts_tablet_manager.cc:1905] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:23.329384 2025 tablet_bootstrap.cc:492] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: Bootstrap starting.
I20260923 20:53:23.331256 2025 tablet_bootstrap.cc:654] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.333118 2047 ts_tablet_manager.cc:1918] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:23.333319 2047 log.cc:1196] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/dc0136c5466b46b1b33740ac892814b6
I20260923 20:53:23.334014 2047 ts_tablet_manager.cc:1939] T dc0136c5466b46b1b33740ac892814b6 P b324980155884832b93867f84c33b62b: Deleting consensus metadata
I20260923 20:53:23.337927 1502 catalog_manager.cc:4841] TS 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3:38661): tablet dc0136c5466b46b1b33740ac892814b6 (table test-table [id=fd5a65f66fea43718e2dd290b3b21f21]) successfully deleted
I20260923 20:53:23.339078 2022 tablet_bootstrap.cc:492] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: No bootstrap required, opened a new log
I20260923 20:53:23.339258 2022 ts_tablet_manager.cc:1397] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260923 20:53:23.340018 2021 tablet_bootstrap.cc:492] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: Bootstrap starting.
I20260923 20:53:23.340150 2022 raft_consensus.cc:348] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.340371 2022 raft_consensus.cc:374] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.340473 2022 raft_consensus.cc:729] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc33d677eaf4fda8ff7e1db22985940, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.340781 2022 consensus_queue.cc:260] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.341439 2022 ts_tablet_manager.cc:1428] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 20:53:23.341638 2021 tablet_bootstrap.cc:654] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.348429 2045 ts_tablet_manager.cc:1918] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:23.348623 2045 log.cc:1196] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/dc0136c5466b46b1b33740ac892814b6
I20260923 20:53:23.348716 1500 catalog_manager.cc:4841] TS b324980155884832b93867f84c33b62b (127.0.97.2:46753): tablet dc0136c5466b46b1b33740ac892814b6 (table test-table [id=fd5a65f66fea43718e2dd290b3b21f21]) successfully deleted
I20260923 20:53:23.349227 2045 ts_tablet_manager.cc:1939] T dc0136c5466b46b1b33740ac892814b6 P bbc33d677eaf4fda8ff7e1db22985940: Deleting consensus metadata
I20260923 20:53:23.349274 2021 tablet_bootstrap.cc:492] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: No bootstrap required, opened a new log
I20260923 20:53:23.349495 2021 ts_tablet_manager.cc:1397] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260923 20:53:23.350462 2021 raft_consensus.cc:348] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.350715 2021 raft_consensus.cc:374] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.350811 2021 raft_consensus.cc:729] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6c34254cee41958872e26c24cbfbcd, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.351130 2021 consensus_queue.cc:260] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.351264 1500 catalog_manager.cc:4841] TS bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1:45505): tablet dc0136c5466b46b1b33740ac892814b6 (table test-table [id=fd5a65f66fea43718e2dd290b3b21f21]) successfully deleted
I20260923 20:53:23.352363 2021 ts_tablet_manager.cc:1428] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:53:23.354656 2025 tablet_bootstrap.cc:492] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: No bootstrap required, opened a new log
I20260923 20:53:23.354830 2025 ts_tablet_manager.cc:1397] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.006s
I20260923 20:53:23.355795 2025 raft_consensus.cc:348] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.356016 2025 raft_consensus.cc:374] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.356122 2025 raft_consensus.cc:729] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b324980155884832b93867f84c33b62b, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.356456 2025 consensus_queue.cc:260] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.358043 2025 ts_tablet_manager.cc:1428] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 20:53:23.370574 2027 raft_consensus.cc:482] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:23.370862 2027 raft_consensus.cc:504] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.372864 2027 leader_election.cc:288] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1:45505), b324980155884832b93867f84c33b62b (127.0.97.2:46753)
I20260923 20:53:23.381111 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.373065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45505 (local address 127.0.97.3:36351)
0923 20:53:23.373383 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.373411 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.373546 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.374418 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.374424 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.374460 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.375051 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.375064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.376345 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.376350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.378468 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.378484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.380273 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.380282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.380480 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.380515 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.380621 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.380773 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260923 20:53:23.381209 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.373535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36351 (local address 127.0.97.1:45505)
0923 20:53:23.373764 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.373774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.373805 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:53:23.373948 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.373957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.374069 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.374208 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.375291 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.376206 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.378667 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.380016 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.380194 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.380327 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.380687 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.380833 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:53:23.381979 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.373939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41719 (local address 127.0.97.2:46753)
0923 20:53:23.374141 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.374147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.375181 (+ 1034us) server_negotiation.cc:407] Connection header received
0923 20:53:23.375401 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.375406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.375498 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.375563 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.376342 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.377113 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.379224 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.380549 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.380706 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.380801 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.381435 (+ 634us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.381632 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:53:23.382673 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.373508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46753 (local address 127.0.97.3:41719)
0923 20:53:23.375030 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.375051 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.375196 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.375752 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.375759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.375784 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.376158 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.376170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.377272 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.377279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.379003 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.379020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.380996 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.381004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.381208 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.381256 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.382181 (+ 925us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.382336 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"thread_start_us":761,"threads_started":1}
I20260923 20:53:23.382472 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb722776df649998ea4cde70c248585" candidate_uuid: "1f6c34254cee41958872e26c24cbfbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc33d677eaf4fda8ff7e1db22985940" is_pre_election: true
I20260923 20:53:23.383006 1672 raft_consensus.cc:2454] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6c34254cee41958872e26c24cbfbcd in term 0.
I20260923 20:53:23.383613 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb722776df649998ea4cde70c248585" candidate_uuid: "1f6c34254cee41958872e26c24cbfbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b324980155884832b93867f84c33b62b" is_pre_election: true
I20260923 20:53:23.383778 1883 leader_election.cc:302] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6c34254cee41958872e26c24cbfbcd, bbc33d677eaf4fda8ff7e1db22985940; no voters:
I20260923 20:53:23.383875 1817 raft_consensus.cc:2454] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f6c34254cee41958872e26c24cbfbcd in term 0.
I20260923 20:53:23.384225 2027 raft_consensus.cc:2781] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:23.384351 2027 raft_consensus.cc:482] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:23.384456 2027 raft_consensus.cc:3037] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.387615 2027 raft_consensus.cc:504] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.388391 2027 leader_election.cc:288] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [CANDIDATE]: Term 1 election: Requested vote from peers bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1:45505), b324980155884832b93867f84c33b62b (127.0.97.2:46753)
I20260923 20:53:23.388881 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb722776df649998ea4cde70c248585" candidate_uuid: "1f6c34254cee41958872e26c24cbfbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc33d677eaf4fda8ff7e1db22985940"
I20260923 20:53:23.389022 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb722776df649998ea4cde70c248585" candidate_uuid: "1f6c34254cee41958872e26c24cbfbcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b324980155884832b93867f84c33b62b"
I20260923 20:53:23.389154 1672 raft_consensus.cc:3037] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.389267 1817 raft_consensus.cc:3037] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.392221 1672 raft_consensus.cc:2454] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6c34254cee41958872e26c24cbfbcd in term 1.
I20260923 20:53:23.392712 1883 leader_election.cc:302] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6c34254cee41958872e26c24cbfbcd, bbc33d677eaf4fda8ff7e1db22985940; no voters:
I20260923 20:53:23.393028 2027 raft_consensus.cc:2781] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:23.393476 1817 raft_consensus.cc:2454] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f6c34254cee41958872e26c24cbfbcd in term 1.
I20260923 20:53:23.393532 2027 raft_consensus.cc:686] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 1 LEADER]: Becoming Leader. State: Replica: 1f6c34254cee41958872e26c24cbfbcd, State: Running, Role: LEADER
I20260923 20:53:23.394028 2027 consensus_queue.cc:237] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.397751 1514 catalog_manager.cc:5495] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1f6c34254cee41958872e26c24cbfbcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } health_report { overall_health: HEALTHY } } }
Sep 23 20:53:23 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:23.444566 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.428629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.97.62:34519)
0923 20:53:23.428845 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.428854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.429615 (+ 761us) server_negotiation.cc:407] Connection header received
0923 20:53:23.430288 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.430293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.430369 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.430486 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:23.431754 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.432572 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.433946 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.434410 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.434571 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.438633 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:23.438639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:23.438642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:23.438673 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:23.441417 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:23.442301 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:23.442307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:23.442311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:23.442448 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:23.442881 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:23.442888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:23.442896 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:23.443170 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:23.443338 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.443778 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.444129 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":72}
I20260923 20:53:23.444607 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.428660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34519 (local address 127.0.0.1:36748)
0923 20:53:23.429487 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.429506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.430135 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.430510 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.430514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.431261 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:23.431569 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.431582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.432724 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.432732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.433734 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.433747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.434617 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.434624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.434802 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.435557 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:23.435576 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:23.438402 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:23.441593 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:23.441608 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:23.441617 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:23.442160 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:23.442580 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:23.442585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:23.442589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:23.442756 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:23.443329 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:23.443338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:23.443505 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.443688 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.444006 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":474,"threads_started":1}
I20260923 20:53:23.451249 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:53:23.452452 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:23.460657 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef664163a994f5a926ddbef738223b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5074769ac9284a1aa27c4b099c6a586f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:53:23.460798 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef664163a994f5a926ddbef738223b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5074769ac9284a1aa27c4b099c6a586f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:53:23.461053 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef664163a994f5a926ddbef738223b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5074769ac9284a1aa27c4b099c6a586f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:53:23.461305 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef664163a994f5a926ddbef738223b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.461512 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef664163a994f5a926ddbef738223b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.461560 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef664163a994f5a926ddbef738223b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:23.468812 2022 tablet_bootstrap.cc:492] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: Bootstrap starting.
I20260923 20:53:23.469383 2021 tablet_bootstrap.cc:492] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: Bootstrap starting.
I20260923 20:53:23.471176 2022 tablet_bootstrap.cc:654] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.471633 2021 tablet_bootstrap.cc:654] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.472807 2025 tablet_bootstrap.cc:492] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: Bootstrap starting.
I20260923 20:53:23.475273 2025 tablet_bootstrap.cc:654] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:23.480626 2025 tablet_bootstrap.cc:492] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: No bootstrap required, opened a new log
I20260923 20:53:23.480888 2025 ts_tablet_manager.cc:1397] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260923 20:53:23.481899 2025 raft_consensus.cc:348] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.482116 2025 raft_consensus.cc:374] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.482167 2025 raft_consensus.cc:729] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b324980155884832b93867f84c33b62b, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.482532 2025 consensus_queue.cc:260] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.483042 2031 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.483363 2022 tablet_bootstrap.cc:492] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: No bootstrap required, opened a new log
I20260923 20:53:23.483409 2031 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:23.483564 2022 ts_tablet_manager.cc:1397] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260923 20:53:23.484367 2022 raft_consensus.cc:348] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.484628 2022 raft_consensus.cc:374] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.484715 2022 raft_consensus.cc:729] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc33d677eaf4fda8ff7e1db22985940, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.485008 2022 consensus_queue.cc:260] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.485653 2021 tablet_bootstrap.cc:492] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: No bootstrap required, opened a new log
I20260923 20:53:23.485543 2032 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.485981 2021 ts_tablet_manager.cc:1397] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20260923 20:53:23.486557 2025 ts_tablet_manager.cc:1428] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260923 20:53:23.486835 2032 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:23.486884 2021 raft_consensus.cc:348] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.487360 2021 raft_consensus.cc:374] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:23.487371 2022 ts_tablet_manager.cc:1428] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260923 20:53:23.487488 2021 raft_consensus.cc:729] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6c34254cee41958872e26c24cbfbcd, State: Initialized, Role: FOLLOWER
I20260923 20:53:23.487778 2021 consensus_queue.cc:260] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.488441 2021 ts_tablet_manager.cc:1428] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:53:23.488287 2027 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.489538 2027 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 20:53:23.534271 1723 tablet.cc:2369] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:53:23.597330 1863 tablet.cc:2369] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:53:23.623395 2031 raft_consensus.cc:482] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:23.623673 2031 raft_consensus.cc:504] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.625751 2031 leader_election.cc:288] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1:45505), 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3:38661)
I20260923 20:53:23.636430 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.626536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38449 (local address 127.0.97.3:38661)
0923 20:53:23.626718 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.626724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.628498 (+ 1774us) server_negotiation.cc:407] Connection header received
0923 20:53:23.628840 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.628848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.628949 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.629052 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.629727 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.630759 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.633557 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.635271 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.635437 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.635554 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.635978 (+ 424us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.636124 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:53:23.636564 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.626537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38661 (local address 127.0.97.2:38449)
0923 20:53:23.628366 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.628391 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.628649 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.629121 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.629129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.629159 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.629513 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.629527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.630913 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.630921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.633307 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.633333 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.635563 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.635570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.635758 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.635810 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.635943 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.636105 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1691,"mutex_wait_us":4,"thread_start_us":1963,"threads_started":1}
I20260923 20:53:23.637064 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.626214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39995 (local address 127.0.97.1:45505)
0923 20:53:23.626454 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.626464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.628982 (+ 2518us) server_negotiation.cc:407] Connection header received
0923 20:53:23.629249 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.629261 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.629322 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.629383 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:23.630286 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.631457 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.633987 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.635318 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.635526 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.635634 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.636239 (+ 605us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.636402 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 20:53:23.637270 1956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef664163a994f5a926ddbef738223b2" candidate_uuid: "b324980155884832b93867f84c33b62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6c34254cee41958872e26c24cbfbcd" is_pre_election: true
I20260923 20:53:23.637466 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.626577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45505 (local address 127.0.97.2:39995)
0923 20:53:23.628840 (+ 2263us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.628858 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.629011 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.629557 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.629560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.629584 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:23.630074 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.630088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.631638 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.631644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.633757 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.633768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.635750 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.635756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.636018 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.636053 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.636798 (+ 745us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.636969 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2126,"mutex_wait_us":88,"thread_start_us":1485,"threads_started":1}
I20260923 20:53:23.637598 1956 raft_consensus.cc:2454] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b324980155884832b93867f84c33b62b in term 0.
I20260923 20:53:23.638165 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef664163a994f5a926ddbef738223b2" candidate_uuid: "b324980155884832b93867f84c33b62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc33d677eaf4fda8ff7e1db22985940" is_pre_election: true
I20260923 20:53:23.638175 1745 leader_election.cc:302] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6c34254cee41958872e26c24cbfbcd, b324980155884832b93867f84c33b62b; no voters:
I20260923 20:53:23.638437 1672 raft_consensus.cc:2454] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b324980155884832b93867f84c33b62b in term 0.
I20260923 20:53:23.638659 2031 raft_consensus.cc:2781] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:23.638818 2031 raft_consensus.cc:482] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:23.638929 2031 raft_consensus.cc:3037] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.641290 2031 raft_consensus.cc:504] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.642078 2031 leader_election.cc:288] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [CANDIDATE]: Term 1 election: Requested vote from peers bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1:45505), 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3:38661)
I20260923 20:53:23.642645 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef664163a994f5a926ddbef738223b2" candidate_uuid: "b324980155884832b93867f84c33b62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc33d677eaf4fda8ff7e1db22985940"
I20260923 20:53:23.642958 1956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef664163a994f5a926ddbef738223b2" candidate_uuid: "b324980155884832b93867f84c33b62b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6c34254cee41958872e26c24cbfbcd"
I20260923 20:53:23.643194 1956 raft_consensus.cc:3037] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.643324 1672 raft_consensus.cc:3037] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:23.645454 1956 raft_consensus.cc:2454] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b324980155884832b93867f84c33b62b in term 1.
I20260923 20:53:23.646065 1745 leader_election.cc:302] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6c34254cee41958872e26c24cbfbcd, b324980155884832b93867f84c33b62b; no voters:
I20260923 20:53:23.646409 2031 raft_consensus.cc:2781] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:23.646610 1672 raft_consensus.cc:2454] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b324980155884832b93867f84c33b62b in term 1.
I20260923 20:53:23.647049 2031 raft_consensus.cc:686] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [term 1 LEADER]: Becoming Leader. State: Replica: b324980155884832b93867f84c33b62b, State: Running, Role: LEADER
I20260923 20:53:23.647534 2031 consensus_queue.cc:237] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } }
I20260923 20:53:23.649845 2065 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: TxnStatusTablet state changed. Reason: New leader b324980155884832b93867f84c33b62b. Latest consensus state: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.650337 2065 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:53:23.652114 2067 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:53:23.652386 2067 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:53:23.652489 1515 catalog_manager.cc:5495] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b reported cstate change: term changed from 0 to 1, leader changed from <none> to b324980155884832b93867f84c33b62b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:23.716646 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.708172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.0.97.2:46753)
0923 20:53:23.708346 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.708353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.708374 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:53:23.709218 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.709223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.709341 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.709512 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:23.710011 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.711271 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.713365 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.714156 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.714304 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.715064 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.715909 (+ 845us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.716057 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.716221 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.716319 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:53:23.716686 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.708018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46753 (local address 127.0.0.1:55956)
0923 20:53:23.708270 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.708294 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.708904 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.709562 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.709568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.709582 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:53:23.709804 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.709864 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.711476 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.711481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.713169 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.713191 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.714378 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.714384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.714660 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.714825 (+ 165us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.716037 (+ 1212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.716078 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.716184 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.716358 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:53:23.721846 1956 raft_consensus.cc:1261] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Refusing update from remote peer b324980155884832b93867f84c33b62b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:23.722335 1672 raft_consensus.cc:1261] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Refusing update from remote peer b324980155884832b93867f84c33b62b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:23.722781 2065 consensus_queue.cc:1035] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:23.724078 2065 consensus_queue.cc:1035] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:23.728161 2032 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: TxnStatusTablet state changed. Reason: New leader b324980155884832b93867f84c33b62b. Latest consensus state: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.728616 2032 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:23.732513 2031 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.732906 2031 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:53:23.733664 2037 mvcc.cc:205] Tried to move back new op lower bound from 7332646108036837376 to 7332646107745030144. Current Snapshot: MvccSnapshot[applied={T|T < 7332646108036837376 or (T in {7332646108036837376})}]
I20260923 20:53:23.736260 2035 mvcc.cc:205] Tried to move back new op lower bound from 7332646108036837376 to 7332646107745030144. Current Snapshot: MvccSnapshot[applied={T|T < 7332646108036837376}]
I20260923 20:53:23.736174 2032 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.736508 2032 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P bbc33d677eaf4fda8ff7e1db22985940: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:23.740139 2027 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: TxnStatusTablet state changed. Reason: New leader b324980155884832b93867f84c33b62b. Latest consensus state: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.740446 2027 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:23.745513 2065 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.745844 2065 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P b324980155884832b93867f84c33b62b: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:53:23.751941 2027 tablet_replica.cc:439] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b324980155884832b93867f84c33b62b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 } } peers { permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 } } peers { permanent_uuid: "1f6c34254cee41958872e26c24cbfbcd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38661 } } }
I20260923 20:53:23.752276 2027 tablet_replica.cc:442] T 6ef664163a994f5a926ddbef738223b2 P 1f6c34254cee41958872e26c24cbfbcd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:23.769543 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.761185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.0.97.3:38661)
0923 20:53:23.761398 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.761408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.761932 (+ 524us) server_negotiation.cc:407] Connection header received
0923 20:53:23.762297 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.762304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.762424 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.762597 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:23.763124 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.764238 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.766438 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.767163 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.767301 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.767847 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.768602 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.768765 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.768914 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.769023 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:53:23.821593 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.799888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42527 (local address 127.0.97.2:46753)
0923 20:53:23.800077 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.800083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.814544 (+ 14461us) server_negotiation.cc:407] Connection header received
0923 20:53:23.814754 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.814763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.814866 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.814987 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:23.815761 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.816890 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.819156 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.819927 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.820040 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.820763 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.820923 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.820986 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.821218 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.821312 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:53:23.822458 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.800245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46753 (local address 127.0.97.3:42527)
0923 20:53:23.814358 (+ 14113us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.814389 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.814594 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.815180 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.815186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.815202 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:53:23.815512 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.815524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.817111 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.817118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.818955 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.818972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.820253 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.820260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.820446 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.820529 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.821068 (+ 539us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.821078 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.821897 (+ 819us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.822057 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13880,"thread_start_us":13721,"threads_started":1}
I20260923 20:53:23.823418 2027 consensus_queue.cc:1035] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b324980155884832b93867f84c33b62b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:23.836434 1671 raft_consensus.cc:1261] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Refusing update from remote peer 1f6c34254cee41958872e26c24cbfbcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:23.837198 2027 consensus_queue.cc:1035] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc33d677eaf4fda8ff7e1db22985940" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:23.901012 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.894151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42679 (local address 127.0.97.3:38661)
0923 20:53:23.894383 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.894392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.895061 (+ 669us) server_negotiation.cc:407] Connection header received
0923 20:53:23.895364 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.895369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.895439 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.895545 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:23.896106 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.897135 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.899083 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.899515 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.899675 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.900131 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.900252 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.900353 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:23.900605 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 20:53:23.900709 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:53:23.901686 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.893677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38661 (local address 127.0.97.2:42679)
0923 20:53:23.894949 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:23.894972 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:23.895198 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:23.895564 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:23.895568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:23.895579 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:53:23.895929 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.895938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.897268 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.897294 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.898916 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:23.898930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.899712 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:23.899719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:23.899866 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:23.899931 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:53:23.900408 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:53:23.900420 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:53:23.900575 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 20:53:23.900747 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":884,"threads_started":1}
I20260923 20:53:23.952099 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:23.952389 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:23.956926 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b74ad04dc444439587d6d12c15919825: Sending DeleteTablet for 3 replicas of tablet bfb722776df649998ea4cde70c248585
I20260923 20:53:23.958231 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb722776df649998ea4cde70c248585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58314
I20260923 20:53:23.958930 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb722776df649998ea4cde70c248585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54078
I20260923 20:53:23.960675 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb722776df649998ea4cde70c248585 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55950
I20260923 20:53:23.977514 2086 tablet_replica.cc:330] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: stopping tablet replica
I20260923 20:53:23.977993 2086 raft_consensus.cc:2229] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:23.978766 2086 raft_consensus.cc:2258] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:23.979265 2088 tablet_replica.cc:330] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: stopping tablet replica
I20260923 20:53:23.980259 2086 ts_tablet_manager.cc:1905] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:23.979494 2088 raft_consensus.cc:2229] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:23.981329 2088 raft_consensus.cc:2258] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:23.985561 2087 tablet_replica.cc:330] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: stopping tablet replica
I20260923 20:53:23.986927 2088 ts_tablet_manager.cc:1905] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:23.991114 2087 raft_consensus.cc:2229] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:23.991457 2087 raft_consensus.cc:2258] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:23.992957 2086 ts_tablet_manager.cc:1918] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:23.993239 2086 log.cc:1196] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/bfb722776df649998ea4cde70c248585
I20260923 20:53:23.994275 2086 ts_tablet_manager.cc:1939] T bfb722776df649998ea4cde70c248585 P 1f6c34254cee41958872e26c24cbfbcd: Deleting consensus metadata
I20260923 20:53:23.995208 2087 ts_tablet_manager.cc:1905] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:23.995931 1502 catalog_manager.cc:4841] TS 1f6c34254cee41958872e26c24cbfbcd (127.0.97.3:38661): tablet bfb722776df649998ea4cde70c248585 (table test-table [id=775744b02f414a1db54e2e686db26783]) successfully deleted
I20260923 20:53:24.008551 2087 ts_tablet_manager.cc:1918] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:24.008759 2087 log.cc:1196] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/bfb722776df649998ea4cde70c248585
I20260923 20:53:24.009164 2087 ts_tablet_manager.cc:1939] T bfb722776df649998ea4cde70c248585 P bbc33d677eaf4fda8ff7e1db22985940: Deleting consensus metadata
I20260923 20:53:24.010459 1500 catalog_manager.cc:4841] TS bbc33d677eaf4fda8ff7e1db22985940 (127.0.97.1:45505): tablet bfb722776df649998ea4cde70c248585 (table test-table [id=775744b02f414a1db54e2e686db26783]) successfully deleted
I20260923 20:53:24.014140 2088 ts_tablet_manager.cc:1918] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:24.014459 2088 log.cc:1196] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/bfb722776df649998ea4cde70c248585
I20260923 20:53:24.015043 2088 ts_tablet_manager.cc:1939] T bfb722776df649998ea4cde70c248585 P b324980155884832b93867f84c33b62b: Deleting consensus metadata
Sep 23 20:53:24 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:24.016521 1500 catalog_manager.cc:4841] TS b324980155884832b93867f84c33b62b (127.0.97.2:46753): tablet bfb722776df649998ea4cde70c248585 (table test-table [id=775744b02f414a1db54e2e686db26783]) successfully deleted
I20260923 20:53:24.025588 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:23.977352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.0.97.1:45505)
0923 20:53:23.977555 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:23.977562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:23.990090 (+ 12528us) server_negotiation.cc:407] Connection header received
0923 20:53:23.990456 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:23.990465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:23.990602 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:23.990715 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:23.998435 (+ 7720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:23.999614 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.010252 (+ 10638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.010974 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.011095 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:24.016811 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:24.016831 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:24.016852 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:24.016902 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:24.021569 (+ 4667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.022718 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.022771 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.022775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.022976 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.023487 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.023492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.023496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.023972 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:24.024298 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:24.024926 (+ 628us) server_negotiation.cc:299] Negotiation successful
0923 20:53:24.025203 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 20:53:24.027731 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54092
I20260923 20:53:24.069542 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:24.031761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.0.97.62:34519)
0923 20:53:24.050478 (+ 18717us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:24.050486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:24.050516 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:53:24.050600 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:24.050605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:24.050724 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:24.050875 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:24.053156 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.054490 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.056495 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.057163 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.057305 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:24.061204 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:24.061211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:24.061216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:24.061266 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:24.065120 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.066139 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.066145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.066149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.066322 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.066830 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.066835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.066839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.067219 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:24.067455 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:24.068859 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0923 20:53:24.069124 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18564,"thread_start_us":18421,"threads_started":1}
W20260923 20:53:24.070179 1515 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36754
I20260923 20:53:24.073138 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1586
I20260923 20:53:24.096201 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:24.075960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.0.97.1:45505)
0923 20:53:24.076150 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:24.076156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:24.078393 (+ 2237us) server_negotiation.cc:407] Connection header received
0923 20:53:24.078591 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:24.078598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:24.078695 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:24.078815 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:24.080427 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.081533 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.083702 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.084425 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.084598 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:24.088487 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:24.088495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:24.088500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:24.088554 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:24.092479 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.093523 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.093532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.093536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.093745 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.094318 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.094327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.094332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.094724 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:24.095039 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:24.095368 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 20:53:24.095631 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 20:53:24.096872 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54098
W20260923 20:53:24.098004 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:24.098225 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 1586
I20260923 20:53:24.130282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1726
Sep 23 20:53:24 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:53:24.153044 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:24.131277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.0.97.2:46753)
0923 20:53:24.131470 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:24.131477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:24.131499 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:53:24.131655 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:24.131660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:24.131758 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:24.131877 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:24.133466 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.134829 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.137034 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.137778 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.138276 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:24.143961 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:24.143983 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:24.144005 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:24.144063 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:24.148416 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.149493 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.149504 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.149509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.149703 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.150222 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.150228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.150233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.150778 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:24.151130 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:24.152367 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0923 20:53:24.152609 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260923 20:53:24.154138 1857 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55960
W20260923 20:53:24.154946 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1726: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:24.155164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 1726
I20260923 20:53:24.185931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1866
Sep 23 20:53:24 dist-test-slave-plcg krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:53:24.206475 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:24.186750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.0.97.3:38661)
0923 20:53:24.186934 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:24.186939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:24.186995 (+ 56us) server_negotiation.cc:407] Connection header received
0923 20:53:24.187222 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:24.187226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:24.187351 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:24.187447 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:24.189208 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.190266 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.192323 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:24.192767 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:24.192877 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:24.198503 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:24.198525 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:24.198548 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:24.198602 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:24.202943 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.203846 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.203857 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.203861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.204055 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:24.204492 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:24.204497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:24.204500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:24.204995 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:24.205320 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:24.205786 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 20:53:24.206071 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:53:24.207324 1997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58330
W20260923 20:53:24.208142 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1866: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:24.208320 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 1866
I20260923 20:53:24.227097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 1483
I20260923 20:53:24.228346 1555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:24.341722 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 1483
2026-09-23T20:53:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4957 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:24 dist-test-slave-plcg krb5kdc[2098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:24 dist-test-slave-plcg krb5kdc[2098](info): set up 2 sockets
Sep 23 20:53:24 dist-test-slave-plcg krb5kdc[2098](info): commencing operation
krb5kdc: starting...
W20260923 20:53:25.438241 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:25 dist-test-slave-plcg krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:25.637673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33405
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:25.995291 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:25.995671 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:25.995849 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:26.003338 2111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:26.003520 2111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:53:26.003600 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:26.003705 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:26.003803 2111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:26.003894 2111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:26.014245 2111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33405
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:26.015635 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:26.017066 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:26.030035 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:26.030022 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:26.030525 2111 server_base.cc:1034] running on GCE node
W20260923 20:53:26.029984 2119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:26.031649 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:26.033246 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:26.034504 2111 hybrid_clock.cc:648] HybridClock initialized: now 1790196806034448 us; error 40 us; skew 500 ppm
I20260923 20:53:26.034873 2111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:26 dist-test-slave-plcg krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:26.038290 2111 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:26.039839 2111 webserver.cc:460] Webserver started at http://127.0.97.62:41903/ using document root <none> and password file <none>
I20260923 20:53:26.040603 2111 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:26.040750 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:26.041077 2111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:26.045265 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "0db54a08269841899d294dee13d0063b"
format_stamp: "Formatted at 2026-09-23 20:53:26 on dist-test-slave-plcg"
server_key: "5522fd93c82389c608ae103a6ac95310"
server_key_iv: "f8706a553bd9a7ef29356537ef6d9f33"
server_key_version: "encryptionkey@0"
I20260923 20:53:26.046273 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "0db54a08269841899d294dee13d0063b"
format_stamp: "Formatted at 2026-09-23 20:53:26 on dist-test-slave-plcg"
server_key: "5522fd93c82389c608ae103a6ac95310"
server_key_iv: "f8706a553bd9a7ef29356537ef6d9f33"
server_key_version: "encryptionkey@0"
I20260923 20:53:26.051884 2111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:26.055881 2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:26.057421 2111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:53:26.057646 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "0db54a08269841899d294dee13d0063b"
format_stamp: "Formatted at 2026-09-23 20:53:26 on dist-test-slave-plcg"
server_key: "5522fd93c82389c608ae103a6ac95310"
server_key_iv: "f8706a553bd9a7ef29356537ef6d9f33"
server_key_version: "encryptionkey@0"
I20260923 20:53:26.057960 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:26.078766 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:26.083649 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:26.084003 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:26.107365 2111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33405
I20260923 20:53:26.107372 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33405 every 8 connection(s)
I20260923 20:53:26.108803 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:26.112846 2188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:26.117367 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 2111
I20260923 20:53:26.117589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:26.118429 388 external_mini_cluster.cc:1428] Setting key 7f08d7b9e209a3ec22843a1040e3793a
I20260923 20:53:26.125288 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Bootstrap starting.
I20260923 20:53:26.129215 2188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:26.130577 2188 log.cc:824] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:26.134268 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: No bootstrap required, opened a new log
Sep 23 20:53:26 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:26.142283 2188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db54a08269841899d294dee13d0063b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33405 } }
I20260923 20:53:26.142709 2188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:26.142843 2188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db54a08269841899d294dee13d0063b, State: Initialized, Role: FOLLOWER
I20260923 20:53:26.143306 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:26.120634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.97.62:33405)
0923 20:53:26.121446 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:26.121462 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:26.121508 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:53:26.123668 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:26.123694 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:26.124157 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:26.124526 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:26.126441 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.127792 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.129599 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.130472 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.130628 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:26.135355 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:26.135363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:26.135384 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:26.135422 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:26.138722 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:26.139529 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:26.139537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:26.139543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:26.139723 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:26.140115 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:26.140121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:26.140126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:26.140737 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:26.141053 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:26.141545 (+ 492us) server_negotiation.cc:299] Negotiation successful
0923 20:53:26.142006 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":389,"threads_started":1}
I20260923 20:53:26.143493 2188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db54a08269841899d294dee13d0063b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33405 } }
I20260923 20:53:26.143781 2188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:26.143936 2188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:26.144158 2188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:26.146690 2188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db54a08269841899d294dee13d0063b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33405 } }
I20260923 20:53:26.147235 2188 leader_election.cc:302] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0db54a08269841899d294dee13d0063b; no voters:
I20260923 20:53:26.147861 2188 leader_election.cc:288] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:26.147943 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:26.149313 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [term 1 LEADER]: Becoming Leader. State: Replica: 0db54a08269841899d294dee13d0063b, State: Running, Role: LEADER
I20260923 20:53:26.149917 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db54a08269841899d294dee13d0063b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33405 } }
I20260923 20:53:26.150213 2188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:26.152611 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0db54a08269841899d294dee13d0063b. Latest consensus state: current_term: 1 leader_uuid: "0db54a08269841899d294dee13d0063b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db54a08269841899d294dee13d0063b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33405 } } }
I20260923 20:53:26.152973 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:26.153568 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0db54a08269841899d294dee13d0063b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0db54a08269841899d294dee13d0063b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33405 } } }
I20260923 20:53:26.153867 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:26.154038 2202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:26.162142 2202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:26.171358 2202 catalog_manager.cc:1347] Generated new cluster ID: 7a99cb1b613b4df4a893b4654b7d622b
I20260923 20:53:26.171494 2202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:26.189922 2202 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:26.191334 2202 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:26.204670 2202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Generated new TSK 0
I20260923 20:53:26.205672 2202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:26.255923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33405
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:26.602115 2214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:26.602415 2214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:26.602520 2214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:26.610034 2214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:26.610152 2214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:26.610240 2214 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:53:26.610291 2214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:53:26.610354 2214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:26.620452 2214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:26.621803 2214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:26.623178 2214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:26.637104 2219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:26.637146 2222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:26.637061 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:26.637903 2214 server_base.cc:1034] running on GCE node
I20260923 20:53:26.638525 2214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:26.639496 2214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:26.640739 2214 hybrid_clock.cc:648] HybridClock initialized: now 1790196806640691 us; error 49 us; skew 500 ppm
I20260923 20:53:26.641085 2214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:26 dist-test-slave-plcg krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:26.644375 2214 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:26.646080 2214 webserver.cc:460] Webserver started at http://127.0.97.1:42017/ using document root <none> and password file <none>
I20260923 20:53:26.646823 2214 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:26.646965 2214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:26.647305 2214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:26.650446 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "924b89b49cba4646a8f869620992fbd0"
format_stamp: "Formatted at 2026-09-23 20:53:26 on dist-test-slave-plcg"
server_key: "a9c9f463e20d5dad5c3c112e9b37453e"
server_key_iv: "8219480f1125f8dfac7117758764c49a"
server_key_version: "encryptionkey@0"
I20260923 20:53:26.651404 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "924b89b49cba4646a8f869620992fbd0"
format_stamp: "Formatted at 2026-09-23 20:53:26 on dist-test-slave-plcg"
server_key: "a9c9f463e20d5dad5c3c112e9b37453e"
server_key_iv: "8219480f1125f8dfac7117758764c49a"
server_key_version: "encryptionkey@0"
I20260923 20:53:26.657171 2214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:53:26.660862 2229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:26.662503 2214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:53:26.662686 2214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "924b89b49cba4646a8f869620992fbd0"
format_stamp: "Formatted at 2026-09-23 20:53:26 on dist-test-slave-plcg"
server_key: "a9c9f463e20d5dad5c3c112e9b37453e"
server_key_iv: "8219480f1125f8dfac7117758764c49a"
server_key_version: "encryptionkey@0"
I20260923 20:53:26.662882 2214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:26.692622 2214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:26.697284 2214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:26.697623 2214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:26.700728 2214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:26.700881 2214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:26.701017 2214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:26.701112 2214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:53:26 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:26.726498 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:26.701121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48595 (local address 127.0.97.62:33405)
0923 20:53:26.701604 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:26.701610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:26.703047 (+ 1437us) server_negotiation.cc:407] Connection header received
0923 20:53:26.706283 (+ 3236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:26.706291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:26.706400 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:26.707362 (+ 962us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:26.709049 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.710170 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.712927 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.713345 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.713496 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:26.719041 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:26.719046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:26.719056 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:26.719094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:26.722262 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:26.723669 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:26.723678 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:26.723684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:26.723884 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:26.724581 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:26.724590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:26.724596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:26.724894 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:26.725075 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:26.725883 (+ 808us) server_negotiation.cc:299] Negotiation successful
0923 20:53:26.726130 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":248,"threads_started":1}
I20260923 20:53:26.727671 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:26.701882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33405 (local address 127.0.97.1:48595)
0923 20:53:26.702817 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:26.702888 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:26.705797 (+ 2909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:26.706897 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:26.706917 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:26.707728 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:26.708804 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:26.708841 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.710371 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.710377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:26.712691 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:26.712707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.713637 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.713645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:26.714217 (+ 572us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:26.715245 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:26.715271 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:26.718745 (+ 3474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:26.722482 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:26.722493 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:26.722517 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:26.723438 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:26.724046 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:26.724053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:26.724059 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:26.724427 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:26.725049 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:26.725059 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:26.725460 (+ 401us) client_negotiation.cc:769] Sending connection context
0923 20:53:26.725750 (+ 290us) client_negotiation.cc:240] Negotiation successful
0923 20:53:26.726466 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":351,"threads_started":1}
I20260923 20:53:26.753427 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:26.734102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33405 (local address 127.0.97.1:60545)
0923 20:53:26.735339 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:26.735364 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:26.735577 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:26.736265 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:26.736273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:26.737092 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:26.737605 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:26.737630 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.739300 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.739313 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:26.741898 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:26.741930 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.743211 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.743218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:26.743392 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:26.744136 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:26.744162 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:26.745741 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:26.749184 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:26.749193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:26.749201 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:26.750015 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:26.750633 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:26.750641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:26.750645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:26.750974 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:26.751904 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:26.751909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:26.752188 (+ 279us) client_negotiation.cc:769] Sending connection context
0923 20:53:26.752388 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 20:53:26.752900 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":833,"threads_started":1}
I20260923 20:53:26.754565 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:26.734107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60545 (local address 127.0.97.62:33405)
0923 20:53:26.734312 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:26.734320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:26.735485 (+ 1165us) server_negotiation.cc:407] Connection header received
0923 20:53:26.735954 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:26.735961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:26.736064 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:26.736185 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:26.737859 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.739097 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.742133 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:26.742934 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:26.743111 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:26.746165 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:26.746174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:26.746180 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:26.746228 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:26.748961 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:26.750203 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:26.750211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:26.750217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:26.750436 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:26.751153 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:26.751161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:26.751167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:26.751651 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:26.751826 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:26.753800 (+ 1974us) server_negotiation.cc:299] Negotiation successful
0923 20:53:26.754114 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:53:26.755187 2214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34225
I20260923 20:53:26.755225 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34225 every 8 connection(s)
I20260923 20:53:26.756649 2214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:26.757164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 2214
I20260923 20:53:26.757402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:26.758240 388 external_mini_cluster.cc:1428] Setting key 83e3de49c827778776163b04b11d6f14
I20260923 20:53:26.760440 2350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33405
I20260923 20:53:26.760694 2350 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:26.761399 2350 heartbeater.cc:511] Master 127.0.97.62:33405 requested a full tablet report, sending...
I20260923 20:53:26.763906 2142 ts_manager.cc:194] Registered new tserver with Master: 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225)
I20260923 20:53:26.766213 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48595
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:26.805625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33405
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:27.161782 2354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:27.162281 2354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:27.162463 2354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:27.170478 2354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:27.170603 2354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:27.170794 2354 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:53:27.170878 2354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:53:27.171015 2354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:27.181195 2354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:27.182657 2354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:27.184105 2354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:27.198777 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:27.198745 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:27.198711 2362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:27.200014 2354 server_base.cc:1034] running on GCE node
I20260923 20:53:27.200713 2354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:27.201749 2354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:27.203017 2354 hybrid_clock.cc:648] HybridClock initialized: now 1790196807202964 us; error 45 us; skew 500 ppm
I20260923 20:53:27.203365 2354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:27 dist-test-slave-plcg krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:27.210862 2354 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:27.212466 2354 webserver.cc:460] Webserver started at http://127.0.97.2:36713/ using document root <none> and password file <none>
I20260923 20:53:27.213156 2354 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:27.213259 2354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:27.213519 2354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:27.216583 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "33b26ffc3320426daed3177d6b7888c3"
format_stamp: "Formatted at 2026-09-23 20:53:27 on dist-test-slave-plcg"
server_key: "77c361dde6017cc950de57d6bff3e43a"
server_key_iv: "93375ddc6af10cfd2e4356a2a9c277ab"
server_key_version: "encryptionkey@0"
I20260923 20:53:27.217363 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "33b26ffc3320426daed3177d6b7888c3"
format_stamp: "Formatted at 2026-09-23 20:53:27 on dist-test-slave-plcg"
server_key: "77c361dde6017cc950de57d6bff3e43a"
server_key_iv: "93375ddc6af10cfd2e4356a2a9c277ab"
server_key_version: "encryptionkey@0"
I20260923 20:53:27.222980 2354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 20:53:27.226680 2369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:27.228111 2354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:53:27.228299 2354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "33b26ffc3320426daed3177d6b7888c3"
format_stamp: "Formatted at 2026-09-23 20:53:27 on dist-test-slave-plcg"
server_key: "77c361dde6017cc950de57d6bff3e43a"
server_key_iv: "93375ddc6af10cfd2e4356a2a9c277ab"
server_key_version: "encryptionkey@0"
I20260923 20:53:27.228541 2354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:27.246217 2354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:27.250617 2354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:27.250931 2354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:27.253925 2354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:27.254086 2354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:27.254237 2354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:27.254335 2354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:53:27 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:27.284183 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.254615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38571 (local address 127.0.97.62:33405)
0923 20:53:27.254848 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.254854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.256683 (+ 1829us) server_negotiation.cc:407] Connection header received
0923 20:53:27.260002 (+ 3319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.260012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.260115 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.260258 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:27.263477 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.264512 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.268149 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.268750 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.268907 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.275290 (+ 6383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:27.275300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:27.275308 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:27.275370 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:27.279700 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.280631 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.280640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.280646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.280848 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.281352 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.281360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.281366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.281886 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:27.282074 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.283536 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.283788 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260923 20:53:27.286111 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.255317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33405 (local address 127.0.97.2:38571)
0923 20:53:27.256377 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:27.256430 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:27.259544 (+ 3114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:27.260628 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:27.260645 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:27.261855 (+ 1210us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:27.263175 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.263219 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.264758 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.264768 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.267892 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.267919 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.269320 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.269332 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.269712 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.270748 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:27.270793 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:27.274867 (+ 4074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:27.279917 (+ 5050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.279928 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.279948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.280457 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.280986 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.280989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.280991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.281220 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.282069 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:27.282080 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:27.282553 (+ 473us) client_negotiation.cc:769] Sending connection context
0923 20:53:27.282977 (+ 424us) client_negotiation.cc:240] Negotiation successful
0923 20:53:27.284759 (+ 1782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":496,"threads_started":1}
I20260923 20:53:27.307482 2354 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40769
I20260923 20:53:27.307641 2488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40769 every 8 connection(s)
I20260923 20:53:27.308858 2354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:27.311208 2489 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33405
I20260923 20:53:27.311475 2489 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:27.311663 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.296792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42715 (local address 127.0.97.62:33405)
0923 20:53:27.297022 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.297032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.297879 (+ 847us) server_negotiation.cc:407] Connection header received
0923 20:53:27.298134 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.298142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.298234 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.298380 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:27.299365 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.300101 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.301492 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.302092 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.302239 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.305384 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:27.305388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:27.305392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:27.305425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:27.308499 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.309268 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.309274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.309277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.309408 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.309936 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.309943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.309947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.310395 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:27.310555 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.311063 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.311306 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:53:27.311880 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.296874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33405 (local address 127.0.97.2:42715)
0923 20:53:27.297675 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:27.297703 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:27.297956 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:27.298419 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:27.298424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:27.298906 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:27.299202 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.299213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.300243 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.300249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.301334 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.301348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.302576 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.302586 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.302842 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.303748 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:27.303769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:27.305140 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:27.308660 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.308666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.308670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.309139 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.309529 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.309535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.309538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.309746 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.310537 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:27.310541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:27.310743 (+ 202us) client_negotiation.cc:769] Sending connection context
0923 20:53:27.310926 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 20:53:27.311329 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":406,"threads_started":1}
I20260923 20:53:27.312570 2489 heartbeater.cc:511] Master 127.0.97.62:33405 requested a full tablet report, sending...
I20260923 20:53:27.314100 2142 ts_manager.cc:194] Registered new tserver with Master: 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769)
I20260923 20:53:27.315153 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38571
I20260923 20:53:27.315927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 2354
I20260923 20:53:27.316110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:27.316540 388 external_mini_cluster.cc:1428] Setting key 5de94bf7cc2b56e37af47dfc95d9ce10
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:27.362421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33405
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:27.700973 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:27.701293 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:27.701471 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:27.709080 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:27.709205 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:27.709399 2493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:53:27.709462 2493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:53:27.709596 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:27.720285 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:27.721686 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:27.723163 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:27.737607 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:27.737574 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:27.737629 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:27.738638 2493 server_base.cc:1034] running on GCE node
I20260923 20:53:27.739267 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:27.740223 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:27.741433 2493 hybrid_clock.cc:648] HybridClock initialized: now 1790196807741388 us; error 37 us; skew 500 ppm
I20260923 20:53:27.741791 2493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:27 dist-test-slave-plcg krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:27.745326 2493 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:27.746920 2493 webserver.cc:460] Webserver started at http://127.0.97.3:46507/ using document root <none> and password file <none>
I20260923 20:53:27.747659 2493 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:27.747813 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:27.748152 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:27.751430 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "86316bb9b7684f31b1653972a98a80f1"
format_stamp: "Formatted at 2026-09-23 20:53:27 on dist-test-slave-plcg"
server_key: "ff821e93514cc294f3cce1133a9ef2ee"
server_key_iv: "aee6a6e97c5501adc6c1c4b8973f2a29"
server_key_version: "encryptionkey@0"
I20260923 20:53:27.752324 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "86316bb9b7684f31b1653972a98a80f1"
format_stamp: "Formatted at 2026-09-23 20:53:27 on dist-test-slave-plcg"
server_key: "ff821e93514cc294f3cce1133a9ef2ee"
server_key_iv: "aee6a6e97c5501adc6c1c4b8973f2a29"
server_key_version: "encryptionkey@0"
I20260923 20:53:27.759110 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:53:27.763495 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:27.765013 2493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:53:27.765213 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "86316bb9b7684f31b1653972a98a80f1"
format_stamp: "Formatted at 2026-09-23 20:53:27 on dist-test-slave-plcg"
server_key: "ff821e93514cc294f3cce1133a9ef2ee"
server_key_iv: "aee6a6e97c5501adc6c1c4b8973f2a29"
server_key_version: "encryptionkey@0"
I20260923 20:53:27.765424 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:27.769745 2350 heartbeater.cc:503] Master 127.0.97.62:33405 was elected leader, sending a full tablet report...
I20260923 20:53:27.787623 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:27.792397 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:27.792742 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:27.795454 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:27.795604 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:27.795725 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:27.795809 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:53:27 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:27.820573 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.796324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48149 (local address 127.0.97.62:33405)
0923 20:53:27.796520 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.796525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.798275 (+ 1750us) server_negotiation.cc:407] Connection header received
0923 20:53:27.800652 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.800660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.800765 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.800847 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:27.803661 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.804601 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.808240 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.808849 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.809001 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.814416 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:27.814421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:27.814424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:27.814460 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:27.817143 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.818071 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.818075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.818078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.818216 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.818710 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.818714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.818716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.818996 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:27.819183 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.819951 (+ 768us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.820199 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:53:27.822024 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.797089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33405 (local address 127.0.97.3:48149)
0923 20:53:27.798093 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:27.798148 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:27.800245 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:27.801523 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:27.801538 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:27.802403 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:27.803442 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.803465 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.804853 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.804859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.807958 (+ 3099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.808021 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.809173 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.809187 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.809406 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.810413 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:27.810465 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:27.814092 (+ 3627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:27.817331 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.817342 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.817362 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.817903 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.818342 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.818347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.818351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.818574 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.819146 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:27.819155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:27.819562 (+ 407us) client_negotiation.cc:769] Sending connection context
0923 20:53:27.819846 (+ 284us) client_negotiation.cc:240] Negotiation successful
0923 20:53:27.820433 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":411,"threads_started":1}
I20260923 20:53:27.845454 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.828295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41549 (local address 127.0.97.62:33405)
0923 20:53:27.828487 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.828493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.829322 (+ 829us) server_negotiation.cc:407] Connection header received
0923 20:53:27.829554 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.829559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.829640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.829761 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:27.831585 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.832293 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.834983 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.835578 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.835733 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.838614 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:27.838619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:27.838622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:27.838655 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:27.842153 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.843170 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.843176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.843181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.843373 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.843833 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.843852 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.843856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.844252 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:27.844413 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.844841 (+ 428us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.845076 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:53:27.845568 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.828352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33405 (local address 127.0.97.3:41549)
0923 20:53:27.829185 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:27.829209 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:27.829410 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:27.829931 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:27.829937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:27.830834 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:27.831311 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.831331 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.832435 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.832442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.834738 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.834757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.835851 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.835860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.836120 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.836963 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:27.837005 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:27.838319 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:27.842336 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.842342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.842346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.843011 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.843530 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:27.843537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:27.843543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:27.843700 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:27.844383 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:27.844388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:27.844584 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 20:53:27.844722 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:53:27.845093 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":499,"threads_started":1}
I20260923 20:53:27.854619 2493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36171
I20260923 20:53:27.854676 2627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36171 every 8 connection(s)
I20260923 20:53:27.855962 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:27.858259 2628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33405
I20260923 20:53:27.858523 2628 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:27.859225 2628 heartbeater.cc:511] Master 127.0.97.62:33405 requested a full tablet report, sending...
I20260923 20:53:27.860692 2142 ts_manager.cc:194] Registered new tserver with Master: 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:27.861613 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48149
I20260923 20:53:27.862273 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 2493
I20260923 20:53:27.862501 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:27.862949 388 external_mini_cluster.cc:1428] Setting key d5a834b97b66e8bed9e6cb3910b4d8c4
I20260923 20:53:27.866322 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:53:27 dist-test-slave-plcg krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:53:27 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:27.922398 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.907311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.0.97.62:33405)
0923 20:53:27.907536 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.907540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.908090 (+ 550us) server_negotiation.cc:407] Connection header received
0923 20:53:27.908207 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.908212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.908451 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.908582 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:27.910323 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.911076 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.912429 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.912870 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.912978 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.917120 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:27.917125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:27.917128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:27.917170 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:27.919753 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.920497 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.920501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.920504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.920672 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:27.921014 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:27.921017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:27.921020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:27.921293 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:27.921497 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.921934 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.922132 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260923 20:53:27.926931 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:27.934544 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:27.962316 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.948031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.97.3:36171)
0923 20:53:27.950015 (+ 1984us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.950026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.950159 (+ 133us) server_negotiation.cc:407] Connection header received
0923 20:53:27.950394 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.950406 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.950870 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.951635 (+ 765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:27.952759 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.955291 (+ 2532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.958994 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.960537 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.960672 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.960934 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.961754 (+ 820us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.961971 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1733,"thread_start_us":1530,"threads_started":1}
I20260923 20:53:27.963068 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.947318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36171 (local address 127.0.0.1:56720)
0923 20:53:27.950014 (+ 2696us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:27.950031 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:27.950193 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:27.951277 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:27.951289 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:27.951334 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:27.952416 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.952436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.955565 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.955585 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.958759 (+ 3174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.958777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.961032 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.961037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.961345 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.961430 (+ 85us) client_negotiation.cc:769] Sending connection context
0923 20:53:27.961688 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 20:53:27.962672 (+ 984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2579,"mutex_wait_us":27,"thread_start_us":1835,"threads_started":1}
I20260923 20:53:27.968351 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.946969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40769 (local address 127.0.0.1:37792)
0923 20:53:27.947773 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:27.947825 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:27.948018 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:27.960981 (+ 12963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:27.960988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:27.961025 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:27.961626 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.961640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.963322 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.963330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.965425 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.965443 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.967419 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.967426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.967612 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.967653 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:53:27.967730 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:53:27.967898 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":152,"thread_start_us":327,"threads_started":1}
I20260923 20:53:27.968510 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.949525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.0.1:33154)
0923 20:53:27.958744 (+ 9219us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:27.958776 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:27.958969 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:27.960694 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:27.960700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:27.960729 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:27.961288 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.961300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.963569 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.963574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.965439 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:27.965451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.967702 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.967711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:27.967884 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.967910 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:53:27.967989 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 20:53:27.968126 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8992,"mutex_wait_us":73,"thread_start_us":8729,"threads_started":1}
I20260923 20:53:27.969216 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.950148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.0.97.1:34225)
0923 20:53:27.959978 (+ 9830us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.959987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.960021 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:53:27.960137 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.960145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.960442 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.960684 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:27.961515 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.963271 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.965624 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.967178 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.967322 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.967549 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.968706 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.968927 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9649,"thread_start_us":9523,"threads_started":1}
I20260923 20:53:27.969623 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:27.947216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.0.97.2:40769)
0923 20:53:27.960401 (+ 13185us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:27.960408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:27.960431 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:53:27.960512 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:27.960517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:27.960747 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:27.961293 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:27.961855 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.963163 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.965679 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:27.967178 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:27.968843 (+ 1665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:27.969045 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:27.969142 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 20:53:27.969318 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13035,"thread_start_us":12895,"threads_started":1}
I20260923 20:53:27.973202 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 693b57614e754b9196a85b3e5756aacb (DEFAULT_TABLE table=test-table [id=28ad4548002648f88728f76e4923b915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:27.973202 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 693b57614e754b9196a85b3e5756aacb (DEFAULT_TABLE table=test-table [id=28ad4548002648f88728f76e4923b915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:27.974833 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693b57614e754b9196a85b3e5756aacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:27.974817 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693b57614e754b9196a85b3e5756aacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:27.975919 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 693b57614e754b9196a85b3e5756aacb (DEFAULT_TABLE table=test-table [id=28ad4548002648f88728f76e4923b915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:27.977252 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693b57614e754b9196a85b3e5756aacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:27.990898 2647 tablet_bootstrap.cc:492] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Bootstrap starting.
I20260923 20:53:27.992691 2648 tablet_bootstrap.cc:492] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Bootstrap starting.
I20260923 20:53:27.992693 2649 tablet_bootstrap.cc:492] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Bootstrap starting.
I20260923 20:53:27.996516 2649 tablet_bootstrap.cc:654] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:27.996542 2648 tablet_bootstrap.cc:654] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:27.996551 2647 tablet_bootstrap.cc:654] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:27.997800 2649 log.cc:824] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:27.997874 2648 log.cc:824] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:27.997866 2647 log.cc:824] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:28.001418 2648 tablet_bootstrap.cc:492] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: No bootstrap required, opened a new log
I20260923 20:53:28.001421 2647 tablet_bootstrap.cc:492] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: No bootstrap required, opened a new log
I20260923 20:53:28.001426 2649 tablet_bootstrap.cc:492] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: No bootstrap required, opened a new log
I20260923 20:53:28.001718 2649 ts_tablet_manager.cc:1397] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:53:28.001718 2647 ts_tablet_manager.cc:1397] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260923 20:53:28.001730 2648 ts_tablet_manager.cc:1397] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260923 20:53:28.011201 2648 raft_consensus.cc:348] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.011701 2648 raft_consensus.cc:374] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.011809 2648 raft_consensus.cc:729] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924b89b49cba4646a8f869620992fbd0, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.012564 2648 consensus_queue.cc:260] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.013288 2647 raft_consensus.cc:348] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.013283 2649 raft_consensus.cc:348] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.013767 2647 raft_consensus.cc:374] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.013767 2649 raft_consensus.cc:374] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.013909 2647 raft_consensus.cc:729] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.013924 2649 raft_consensus.cc:729] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86316bb9b7684f31b1653972a98a80f1, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.014621 2649 consensus_queue.cc:260] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.014595 2647 consensus_queue.cc:260] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.016968 2489 heartbeater.cc:503] Master 127.0.97.62:33405 was elected leader, sending a full tablet report...
I20260923 20:53:28.017750 2647 ts_tablet_manager.cc:1428] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260923 20:53:28.018049 2649 ts_tablet_manager.cc:1428] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Time spent starting tablet: real 0.016s user 0.018s sys 0.000s
I20260923 20:53:28.019083 2628 heartbeater.cc:503] Master 127.0.97.62:33405 was elected leader, sending a full tablet report...
I20260923 20:53:28.020440 2648 ts_tablet_manager.cc:1428] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
W20260923 20:53:28.061146 2490 tablet.cc:2369] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:53:28.083097 2654 raft_consensus.cc:482] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:28.083491 2654 raft_consensus.cc:504] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.085777 2654 leader_election.cc:288] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171), 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225)
I20260923 20:53:28.094214 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.086441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35917 (local address 127.0.97.1:34225)
0923 20:53:28.086675 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.086684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.087357 (+ 673us) server_negotiation.cc:407] Connection header received
0923 20:53:28.087775 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.087785 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.087884 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.088007 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:28.088955 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.090115 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.092177 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.092953 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.093133 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.093257 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.093746 (+ 489us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.093911 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:53:28.094259 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.086274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60601 (local address 127.0.97.3:36171)
0923 20:53:28.086560 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.086570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.087858 (+ 1288us) server_negotiation.cc:407] Connection header received
0923 20:53:28.088151 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.088160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.088242 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.088313 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:28.089273 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.090300 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.092172 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.093014 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.093178 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.093297 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.093790 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.093975 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 20:53:28.094378 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.086250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.97.2:35917)
0923 20:53:28.087221 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.087247 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.087499 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.088209 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.088215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.088241 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:28.088730 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.088748 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.090300 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.090304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.091983 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.091996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.093238 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.093249 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.093433 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.093515 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.093717 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.093911 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":124,"thread_start_us":582,"threads_started":1}
I20260923 20:53:28.094673 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.086855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36171 (local address 127.0.97.2:60601)
0923 20:53:28.087735 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.087757 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.087976 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.088567 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.088572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.088607 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:28.089021 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.089035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.090458 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.090464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.091982 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.091996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.093274 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.093281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.093438 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.093485 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.093761 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.093968 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":3,"thread_start_us":500,"threads_started":1}
I20260923 20:53:28.095431 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693b57614e754b9196a85b3e5756aacb" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924b89b49cba4646a8f869620992fbd0" is_pre_election: true
I20260923 20:53:28.095754 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693b57614e754b9196a85b3e5756aacb" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1" is_pre_election: true
I20260923 20:53:28.095969 2305 raft_consensus.cc:2454] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 0.
I20260923 20:53:28.096261 2583 raft_consensus.cc:2454] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 0.
I20260923 20:53:28.096681 2372 leader_election.cc:302] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:28.097126 2654 raft_consensus.cc:2781] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:28.097256 2654 raft_consensus.cc:482] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:28.097337 2654 raft_consensus.cc:3037] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Advancing to term 1
W20260923 20:53:28.107740 2629 tablet.cc:2369] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:53:28.186412 2654 raft_consensus.cc:504] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.187507 2654 leader_election.cc:288] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 election: Requested vote from peers 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171), 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225)
I20260923 20:53:28.188696 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693b57614e754b9196a85b3e5756aacb" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1"
I20260923 20:53:28.188786 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693b57614e754b9196a85b3e5756aacb" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924b89b49cba4646a8f869620992fbd0"
I20260923 20:53:28.188984 2583 raft_consensus.cc:3037] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:28.189033 2305 raft_consensus.cc:3037] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:28.191587 2305 raft_consensus.cc:2454] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 1.
I20260923 20:53:28.191639 2583 raft_consensus.cc:2454] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 1.
I20260923 20:53:28.192216 2372 leader_election.cc:302] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:28.192584 2654 raft_consensus.cc:2781] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:28.193060 2654 raft_consensus.cc:686] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 1 LEADER]: Becoming Leader. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Running, Role: LEADER
I20260923 20:53:28.193742 2654 consensus_queue.cc:237] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } }
I20260923 20:53:28.199306 2142 catalog_manager.cc:5495] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "33b26ffc3320426daed3177d6b7888c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } health_report { overall_health: UNKNOWN } } }
W20260923 20:53:28.261234 2351 tablet.cc:2369] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 20:53:28 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:28.319307 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.301388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.0.97.62:33405)
0923 20:53:28.301648 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.301657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.302552 (+ 895us) server_negotiation.cc:407] Connection header received
0923 20:53:28.302814 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.302820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.302905 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.303031 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:28.304567 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.305490 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.307033 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.307469 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.307625 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.311856 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:28.311861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:28.311865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:28.311899 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:28.315957 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:28.317084 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:28.317092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:28.317097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:28.317289 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:28.317800 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:28.317804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:28.317806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:28.318160 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:28.318291 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.318826 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.319026 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 20:53:28.319715 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.301382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33405 (local address 127.0.0.1:45554)
0923 20:53:28.302369 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.302392 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.302599 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.303165 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.303170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.304057 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:28.304308 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.304318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.305701 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.305706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.306874 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.306887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.307671 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.307678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.307854 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.308663 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:28.308682 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:28.311529 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:28.316173 (+ 4644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:28.316192 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:28.316202 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:28.316891 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:28.317436 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:28.317441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:28.317446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:28.317641 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:28.318304 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:28.318314 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:28.318544 (+ 230us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.318725 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.319221 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":134,"thread_start_us":603,"threads_started":1}
I20260923 20:53:28.326288 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45554:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:53:28.327410 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:28.337110 2280 tablet_service.cc:1467] Processing CreateTablet for tablet dff3ba8f5a024dc18f554ac29b51a804 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bb51335baead4308928cb484c897240a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:53:28.337111 2419 tablet_service.cc:1467] Processing CreateTablet for tablet dff3ba8f5a024dc18f554ac29b51a804 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bb51335baead4308928cb484c897240a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:53:28.337358 2558 tablet_service.cc:1467] Processing CreateTablet for tablet dff3ba8f5a024dc18f554ac29b51a804 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bb51335baead4308928cb484c897240a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:53:28.338115 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff3ba8f5a024dc18f554ac29b51a804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:28.338124 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff3ba8f5a024dc18f554ac29b51a804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:28.338140 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff3ba8f5a024dc18f554ac29b51a804. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:28.344118 2648 tablet_bootstrap.cc:492] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: Bootstrap starting.
I20260923 20:53:28.345800 2648 tablet_bootstrap.cc:654] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:28.346760 2649 tablet_bootstrap.cc:492] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: Bootstrap starting.
I20260923 20:53:28.347330 2647 tablet_bootstrap.cc:492] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: Bootstrap starting.
I20260923 20:53:28.348542 2649 tablet_bootstrap.cc:654] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:28.349051 2648 tablet_bootstrap.cc:492] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: No bootstrap required, opened a new log
I20260923 20:53:28.349244 2648 ts_tablet_manager.cc:1397] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 20:53:28.349288 2647 tablet_bootstrap.cc:654] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:28.350915 2648 raft_consensus.cc:348] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.351229 2648 raft_consensus.cc:374] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.351341 2648 raft_consensus.cc:729] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924b89b49cba4646a8f869620992fbd0, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.351670 2648 consensus_queue.cc:260] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.351979 2649 tablet_bootstrap.cc:492] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: No bootstrap required, opened a new log
I20260923 20:53:28.352185 2649 ts_tablet_manager.cc:1397] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260923 20:53:28.352227 2647 tablet_bootstrap.cc:492] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: No bootstrap required, opened a new log
I20260923 20:53:28.352368 2647 ts_tablet_manager.cc:1397] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 20:53:28.352216 2653 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.352632 2653 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:28.353142 2648 ts_tablet_manager.cc:1428] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:53:28.353029 2649 raft_consensus.cc:348] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.353304 2649 raft_consensus.cc:374] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.353193 2647 raft_consensus.cc:348] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.353432 2647 raft_consensus.cc:374] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.353430 2649 raft_consensus.cc:729] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86316bb9b7684f31b1653972a98a80f1, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.353502 2647 raft_consensus.cc:729] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.353742 2647 consensus_queue.cc:260] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.354305 2647 ts_tablet_manager.cc:1428] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:53:28.354524 2654 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.354884 2654 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:28.353695 2649 consensus_queue.cc:260] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.355688 2649 ts_tablet_manager.cc:1428] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260923 20:53:28.355808 2655 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.356127 2655 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:28.511715 2653 raft_consensus.cc:482] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:28.512032 2653 raft_consensus.cc:504] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.513947 2653 leader_election.cc:288] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:28.521467 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.514359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40743 (local address 127.0.97.2:40769)
0923 20:53:28.515139 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.515146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.515759 (+ 613us) server_negotiation.cc:407] Connection header received
0923 20:53:28.515951 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.515955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.516011 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.516115 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:28.516772 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.517497 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.519601 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.520416 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.520578 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.520701 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.520982 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.521116 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":534,"threads_started":1}
I20260923 20:53:28.521631 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.514360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47261 (local address 127.0.97.3:36171)
0923 20:53:28.514595 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.514605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.515446 (+ 841us) server_negotiation.cc:407] Connection header received
0923 20:53:28.515867 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.515874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.515959 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.516121 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:28.516778 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.517535 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.519745 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.520595 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.520776 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.520890 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.521200 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.521335 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:53:28.521834 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.514354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36171 (local address 127.0.97.1:47261)
0923 20:53:28.515293 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.515318 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.515519 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.516142 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.516147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.516198 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:28.516588 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.516598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.517675 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.517680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.519423 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.519437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.520850 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.520857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.521034 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.521058 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.521173 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.521317 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":26,"spinlock_wait_cycles":7040,"thread_start_us":643,"threads_started":1}
I20260923 20:53:28.521544 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.514705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40769 (local address 127.0.97.1:40743)
0923 20:53:28.515656 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.515672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.515814 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.516139 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.516144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.516183 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:28.516593 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.516603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.517645 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.517651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.519427 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.519440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.520635 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.520641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.520824 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.520848 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.520956 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.521097 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":86,"spinlock_wait_cycles":2304,"thread_start_us":575,"threads_started":1}
I20260923 20:53:28.522657 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff3ba8f5a024dc18f554ac29b51a804" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1" is_pre_election: true
I20260923 20:53:28.522953 2583 raft_consensus.cc:2454] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 0.
I20260923 20:53:28.522956 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff3ba8f5a024dc18f554ac29b51a804" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b26ffc3320426daed3177d6b7888c3" is_pre_election: true
I20260923 20:53:28.523378 2439 raft_consensus.cc:2454] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 0.
I20260923 20:53:28.523551 2231 leader_election.cc:302] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86316bb9b7684f31b1653972a98a80f1, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:28.524050 2653 raft_consensus.cc:2781] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:28.524185 2653 raft_consensus.cc:482] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:28.524298 2653 raft_consensus.cc:3037] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:28.526638 2653 raft_consensus.cc:504] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.527302 2653 leader_election.cc:288] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 election: Requested vote from peers 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:28.527869 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff3ba8f5a024dc18f554ac29b51a804" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b26ffc3320426daed3177d6b7888c3"
I20260923 20:53:28.527917 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff3ba8f5a024dc18f554ac29b51a804" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1"
I20260923 20:53:28.528086 2439 raft_consensus.cc:3037] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:28.528209 2583 raft_consensus.cc:3037] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:28.530422 2583 raft_consensus.cc:2454] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 1.
I20260923 20:53:28.530933 2231 leader_election.cc:302] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86316bb9b7684f31b1653972a98a80f1, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:28.531051 2439 raft_consensus.cc:2454] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 1.
I20260923 20:53:28.531260 2653 raft_consensus.cc:2781] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:28.531786 2653 raft_consensus.cc:686] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [term 1 LEADER]: Becoming Leader. State: Replica: 924b89b49cba4646a8f869620992fbd0, State: Running, Role: LEADER
I20260923 20:53:28.532382 2653 consensus_queue.cc:237] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.535951 2672 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: TxnStatusTablet state changed. Reason: New leader 924b89b49cba4646a8f869620992fbd0. Latest consensus state: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.536451 2672 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:53:28.537163 2674 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:53:28.537436 2674 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:53:28.537669 2142 catalog_manager.cc:5495] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 924b89b49cba4646a8f869620992fbd0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:28.602234 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.595408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.0.1:33162)
0923 20:53:28.595674 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.595697 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.595858 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.596411 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.596418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.596434 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:53:28.596645 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.596657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.597727 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.597733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.599374 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.599391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.600257 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.600264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.600491 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.600590 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:53:28.601568 (+ 978us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:53:28.601603 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.601697 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.601896 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:53:28.602257 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.595458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.0.97.1:34225)
0923 20:53:28.596054 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.596058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.596075 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:53:28.596155 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.596158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.596230 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.596363 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:28.596790 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.597593 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.599574 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.600082 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.600198 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.600829 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:28.601447 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:28.601584 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.601727 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.601803 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":399,"threads_started":1}
I20260923 20:53:28.611222 2439 raft_consensus.cc:1261] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Refusing update from remote peer 924b89b49cba4646a8f869620992fbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:28.611244 2583 raft_consensus.cc:1261] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Refusing update from remote peer 924b89b49cba4646a8f869620992fbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:28.612282 2672 consensus_queue.cc:1035] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:28.612699 2653 consensus_queue.cc:1035] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:28.624086 2655 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: TxnStatusTablet state changed. Reason: New leader 924b89b49cba4646a8f869620992fbd0. Latest consensus state: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.624385 2655 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:28.625013 2654 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: TxnStatusTablet state changed. Reason: New leader 924b89b49cba4646a8f869620992fbd0. Latest consensus state: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.625288 2654 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:28.627812 2678 mvcc.cc:205] Tried to move back new op lower bound from 7332646128060813312 to 7332646127752380416. Current Snapshot: MvccSnapshot[applied={T|T < 7332646128060813312}]
I20260923 20:53:28.629781 2672 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.630100 2672 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:53:28.633332 2654 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.633632 2654 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 33b26ffc3320426daed3177d6b7888c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:28.634120 2672 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.634371 2672 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 924b89b49cba4646a8f869620992fbd0: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:53:28.637907 2655 tablet_replica.cc:439] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } } }
I20260923 20:53:28.638164 2655 tablet_replica.cc:442] T dff3ba8f5a024dc18f554ac29b51a804 P 86316bb9b7684f31b1653972a98a80f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:53:28.648332 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.639971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.0.97.2:40769)
0923 20:53:28.640172 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.640178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.642231 (+ 2053us) server_negotiation.cc:407] Connection header received
0923 20:53:28.642448 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.642452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.642536 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.642663 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:28.643249 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.644001 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.646145 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.646553 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.646654 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.647153 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:28.647705 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:28.647810 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.647945 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.648038 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:53:28.665957 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.659285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.97.2:48789)
0923 20:53:28.660000 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.660022 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.660149 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.660514 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.660517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.660527 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:53:28.660766 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.660774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.662252 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.662256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.663507 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.663521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.664555 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.664560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.664771 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.664845 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:53:28.665336 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:53:28.665356 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.665426 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.665550 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":379,"threads_started":1}
I20260923 20:53:28.665951 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.659317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48789 (local address 127.0.97.1:34225)
0923 20:53:28.659523 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.659530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.660112 (+ 582us) server_negotiation.cc:407] Connection header received
0923 20:53:28.660274 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.660280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.660369 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.660476 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:28.660901 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.662121 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.663665 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.664342 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.664482 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.665069 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:28.665219 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:28.665325 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.665467 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.665552 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:53:28.677330 2583 raft_consensus.cc:1261] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Refusing update from remote peer 33b26ffc3320426daed3177d6b7888c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:28.677685 2305 raft_consensus.cc:1261] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Refusing update from remote peer 33b26ffc3320426daed3177d6b7888c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:28.678107 2658 consensus_queue.cc:1035] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:28.678535 2654 consensus_queue.cc:1035] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:28.685509 2680 mvcc.cc:205] Tried to move back new op lower bound from 7332646128335650816 to 7332646126366355456. Current Snapshot: MvccSnapshot[applied={T|T < 7332646128335650816}]
I20260923 20:53:28.726004 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.718053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52509 (local address 127.0.97.2:40769)
0923 20:53:28.718733 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:28.718740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:28.719315 (+ 575us) server_negotiation.cc:407] Connection header received
0923 20:53:28.719553 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:28.719559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:28.719642 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:28.719742 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:28.720355 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.721401 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.723712 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.724271 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.724428 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.724918 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:28.725035 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:28.725146 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:28.725586 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:53:28.725686 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512}
I20260923 20:53:28.726004 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:28.718213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40769 (local address 127.0.97.1:52509)
0923 20:53:28.719195 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:28.719224 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:28.719404 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:28.719795 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:28.719800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:28.719814 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:53:28.720187 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.720198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.721552 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.721558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.723545 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:28.723559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:28.724464 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:28.724471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:28.724658 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:28.724721 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:53:28.725208 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:53:28.725218 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:53:28.725320 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:53:28.725480 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":675,"threads_started":1}
I20260923 20:53:28.776998 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:28.777400 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:28.781890 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Sending DeleteTablet for 3 replicas of tablet 693b57614e754b9196a85b3e5756aacb
I20260923 20:53:28.783120 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 693b57614e754b9196a85b3e5756aacb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37792
I20260923 20:53:28.789381 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 693b57614e754b9196a85b3e5756aacb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56720
I20260923 20:53:28.790560 2700 tablet_replica.cc:330] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: stopping tablet replica
I20260923 20:53:28.791164 2700 raft_consensus.cc:2229] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:28.792013 2700 raft_consensus.cc:2258] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:28.794324 2700 ts_tablet_manager.cc:1905] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:28.796089 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 693b57614e754b9196a85b3e5756aacb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33154
I20260923 20:53:28.803625 2700 ts_tablet_manager.cc:1918] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:28.803844 2700 log.cc:1196] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/693b57614e754b9196a85b3e5756aacb
I20260923 20:53:28.804505 2701 tablet_replica.cc:330] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: stopping tablet replica
I20260923 20:53:28.804967 2701 raft_consensus.cc:2229] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:28.805375 2701 raft_consensus.cc:2258] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:28.805971 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 20:53:28.806232 2702 tablet_replica.cc:330] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: stopping tablet replica
I20260923 20:53:28.806653 2702 raft_consensus.cc:2229] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 20:53:28.806883 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:28.808707 2701 ts_tablet_manager.cc:1905] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:28.814201 2700 ts_tablet_manager.cc:1939] T 693b57614e754b9196a85b3e5756aacb P 33b26ffc3320426daed3177d6b7888c3: Deleting consensus metadata
I20260923 20:53:28.816733 2129 catalog_manager.cc:4841] TS 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769): tablet 693b57614e754b9196a85b3e5756aacb (table test-table [id=28ad4548002648f88728f76e4923b915]) successfully deleted
I20260923 20:53:28.817265 2701 ts_tablet_manager.cc:1918] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:28.817458 2701 log.cc:1196] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/693b57614e754b9196a85b3e5756aacb
I20260923 20:53:28.817880 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b17b924374b5a91a1727cf13a76eb (DEFAULT_TABLE table=test-table [id=95f63f1d3cde455fb165250e7377db89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:28.818189 2702 raft_consensus.cc:2258] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:28.818893 2701 ts_tablet_manager.cc:1939] T 693b57614e754b9196a85b3e5756aacb P 86316bb9b7684f31b1653972a98a80f1: Deleting consensus metadata
I20260923 20:53:28.820928 2702 ts_tablet_manager.cc:1905] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:28.821441 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b17b924374b5a91a1727cf13a76eb (DEFAULT_TABLE table=test-table [id=95f63f1d3cde455fb165250e7377db89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:28.821966 2128 catalog_manager.cc:4841] TS 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171): tablet 693b57614e754b9196a85b3e5756aacb (table test-table [id=28ad4548002648f88728f76e4923b915]) successfully deleted
I20260923 20:53:28.822196 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b17b924374b5a91a1727cf13a76eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:28.822570 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b17b924374b5a91a1727cf13a76eb (DEFAULT_TABLE table=test-table [id=95f63f1d3cde455fb165250e7377db89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:28.822990 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b17b924374b5a91a1727cf13a76eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:28.826397 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b17b924374b5a91a1727cf13a76eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:28.829761 2647 tablet_bootstrap.cc:492] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: Bootstrap starting.
I20260923 20:53:28.831566 2647 tablet_bootstrap.cc:654] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:28.838407 2649 tablet_bootstrap.cc:492] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: Bootstrap starting.
I20260923 20:53:28.839335 2702 ts_tablet_manager.cc:1918] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:28.839532 2702 log.cc:1196] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/693b57614e754b9196a85b3e5756aacb
I20260923 20:53:28.840312 2649 tablet_bootstrap.cc:654] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:28.840378 2702 ts_tablet_manager.cc:1939] T 693b57614e754b9196a85b3e5756aacb P 924b89b49cba4646a8f869620992fbd0: Deleting consensus metadata
I20260923 20:53:28.842801 2129 catalog_manager.cc:4841] TS 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225): tablet 693b57614e754b9196a85b3e5756aacb (table test-table [id=28ad4548002648f88728f76e4923b915]) successfully deleted
I20260923 20:53:28.844166 2648 tablet_bootstrap.cc:492] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: Bootstrap starting.
I20260923 20:53:28.846690 2648 tablet_bootstrap.cc:654] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:28.849227 2647 tablet_bootstrap.cc:492] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: No bootstrap required, opened a new log
I20260923 20:53:28.849527 2647 ts_tablet_manager.cc:1397] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.010s
I20260923 20:53:28.852484 2647 raft_consensus.cc:348] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.853025 2647 raft_consensus.cc:374] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.853173 2647 raft_consensus.cc:729] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.853595 2647 consensus_queue.cc:260] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.858861 2647 ts_tablet_manager.cc:1428] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20260923 20:53:28.864527 2649 tablet_bootstrap.cc:492] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: No bootstrap required, opened a new log
I20260923 20:53:28.864832 2649 ts_tablet_manager.cc:1397] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.009s
I20260923 20:53:28.865564 2649 raft_consensus.cc:348] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.865796 2649 raft_consensus.cc:374] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.866732 2648 tablet_bootstrap.cc:492] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: No bootstrap required, opened a new log
I20260923 20:53:28.867123 2648 ts_tablet_manager.cc:1397] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.009s
I20260923 20:53:28.868026 2648 raft_consensus.cc:348] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.868389 2648 raft_consensus.cc:374] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:28.868517 2648 raft_consensus.cc:729] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924b89b49cba4646a8f869620992fbd0, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.868870 2648 consensus_queue.cc:260] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.869671 2648 ts_tablet_manager.cc:1428] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:53:28.869863 2649 raft_consensus.cc:729] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86316bb9b7684f31b1653972a98a80f1, State: Initialized, Role: FOLLOWER
I20260923 20:53:28.870281 2649 consensus_queue.cc:260] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:28.873353 2649 ts_tablet_manager.cc:1428] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260923 20:53:29.004784 2653 raft_consensus.cc:482] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:29.005018 2653 raft_consensus.cc:504] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:29.005761 2653 leader_election.cc:288] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:29.006376 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b17b924374b5a91a1727cf13a76eb" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b26ffc3320426daed3177d6b7888c3" is_pre_election: true
I20260923 20:53:29.006655 2439 raft_consensus.cc:2454] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 0.
I20260923 20:53:29.006572 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b17b924374b5a91a1727cf13a76eb" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1" is_pre_election: true
I20260923 20:53:29.006853 2582 raft_consensus.cc:2454] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 0.
I20260923 20:53:29.007243 2232 leader_election.cc:302] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:29.007804 2653 raft_consensus.cc:2781] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:29.007994 2653 raft_consensus.cc:482] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:29.008123 2653 raft_consensus.cc:3037] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:29.011833 2653 raft_consensus.cc:504] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:29.012636 2653 leader_election.cc:288] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 election: Requested vote from peers 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:29.013207 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b17b924374b5a91a1727cf13a76eb" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b26ffc3320426daed3177d6b7888c3"
I20260923 20:53:29.013490 2439 raft_consensus.cc:3037] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:29.013537 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b17b924374b5a91a1727cf13a76eb" candidate_uuid: "924b89b49cba4646a8f869620992fbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1"
I20260923 20:53:29.013764 2582 raft_consensus.cc:3037] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:29.016690 2439 raft_consensus.cc:2454] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 1.
I20260923 20:53:29.017144 2582 raft_consensus.cc:2454] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 924b89b49cba4646a8f869620992fbd0 in term 1.
I20260923 20:53:29.017429 2232 leader_election.cc:302] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:29.017972 2653 raft_consensus.cc:2781] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:29.018204 2653 raft_consensus.cc:686] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 1 LEADER]: Becoming Leader. State: Replica: 924b89b49cba4646a8f869620992fbd0, State: Running, Role: LEADER
I20260923 20:53:29.018561 2653 consensus_queue.cc:237] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:29.021468 2142 catalog_manager.cc:5495] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 924b89b49cba4646a8f869620992fbd0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "924b89b49cba4646a8f869620992fbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:29.056499 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:29.049418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.0.97.1:34225)
0923 20:53:29.049605 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:29.049611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:29.049659 (+ 48us) server_negotiation.cc:407] Connection header received
0923 20:53:29.049887 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:29.049892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:29.049984 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:29.050080 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:29.050558 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:29.051808 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:29.053476 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:29.054234 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:29.054448 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:29.054979 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:29.055159 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:29.055262 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:29.055965 (+ 703us) server_negotiation.cc:299] Negotiation successful
0923 20:53:29.056136 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:53:29.072381 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:29.066320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.97.1:35235)
0923 20:53:29.066505 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:29.066543 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:29.066681 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:29.067048 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:29.067051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:29.067061 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:53:29.067287 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:29.067295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:29.068904 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:29.068908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:29.070120 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:29.070137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:29.071302 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:29.071307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:29.071420 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:29.071459 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:53:29.071920 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:53:29.071929 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:53:29.071991 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 20:53:29.072109 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":29}
I20260923 20:53:29.072481 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:29.066390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35235 (local address 127.0.97.1:34225)
0923 20:53:29.066620 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:29.066627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:29.066655 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:53:29.066799 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:29.066806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:29.066919 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:29.067036 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:29.067438 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:29.068682 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:29.070291 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:29.071127 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:29.071299 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:29.071621 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:29.071798 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:29.071897 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:29.072018 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 20:53:29.072122 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":78}
I20260923 20:53:29.082588 2582 raft_consensus.cc:1261] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Refusing update from remote peer 924b89b49cba4646a8f869620992fbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:29.082649 2439 raft_consensus.cc:1261] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Refusing update from remote peer 924b89b49cba4646a8f869620992fbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:29.083283 2653 consensus_queue.cc:1035] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:29.083632 2672 consensus_queue.cc:1035] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:29.091428 2679 mvcc.cc:205] Tried to move back new op lower bound from 7332646129996845056 to 7332646129742807040. Current Snapshot: MvccSnapshot[applied={T|T < 7332646129996845056}]
W20260923 20:53:29.114205 2678 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:53:30.724606 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:30.724817 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:30.728662 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Sending DeleteTablet for 3 replicas of tablet 7f4b17b924374b5a91a1727cf13a76eb
I20260923 20:53:30.729418 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4b17b924374b5a91a1727cf13a76eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33154
I20260923 20:53:30.729562 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4b17b924374b5a91a1727cf13a76eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37792
I20260923 20:53:30.729585 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4b17b924374b5a91a1727cf13a76eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56720
I20260923 20:53:30.731230 2733 tablet_replica.cc:330] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: stopping tablet replica
I20260923 20:53:30.731464 2733 raft_consensus.cc:2229] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:30.731771 2733 raft_consensus.cc:2258] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:30.732031 2731 tablet_replica.cc:330] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: stopping tablet replica
I20260923 20:53:30.732302 2732 tablet_replica.cc:330] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: stopping tablet replica
I20260923 20:53:30.732336 2731 raft_consensus.cc:2229] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:30.732532 2732 raft_consensus.cc:2229] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:30.732801 2732 raft_consensus.cc:2258] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:30.732976 2733 ts_tablet_manager.cc:1905] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:30.732980 2731 raft_consensus.cc:2258] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:30.733959 2732 ts_tablet_manager.cc:1905] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:30.733959 2731 ts_tablet_manager.cc:1905] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:30.740495 2733 ts_tablet_manager.cc:1918] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:53:30.740736 2733 log.cc:1196] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/7f4b17b924374b5a91a1727cf13a76eb
I20260923 20:53:30.741217 2733 ts_tablet_manager.cc:1939] T 7f4b17b924374b5a91a1727cf13a76eb P 33b26ffc3320426daed3177d6b7888c3: Deleting consensus metadata
I20260923 20:53:30.741966 2731 ts_tablet_manager.cc:1918] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:53:30.742165 2731 log.cc:1196] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/7f4b17b924374b5a91a1727cf13a76eb
I20260923 20:53:30.742261 2732 ts_tablet_manager.cc:1918] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:53:30.742449 2732 log.cc:1196] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/7f4b17b924374b5a91a1727cf13a76eb
I20260923 20:53:30.742604 2129 catalog_manager.cc:4841] TS 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769): tablet 7f4b17b924374b5a91a1727cf13a76eb (table test-table [id=95f63f1d3cde455fb165250e7377db89]) successfully deleted
I20260923 20:53:30.742977 2732 ts_tablet_manager.cc:1939] T 7f4b17b924374b5a91a1727cf13a76eb P 86316bb9b7684f31b1653972a98a80f1: Deleting consensus metadata
I20260923 20:53:30.743271 2731 ts_tablet_manager.cc:1939] T 7f4b17b924374b5a91a1727cf13a76eb P 924b89b49cba4646a8f869620992fbd0: Deleting consensus metadata
I20260923 20:53:30.744026 2128 catalog_manager.cc:4841] TS 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171): tablet 7f4b17b924374b5a91a1727cf13a76eb (table test-table [id=95f63f1d3cde455fb165250e7377db89]) successfully deleted
I20260923 20:53:30.744350 2129 catalog_manager.cc:4841] TS 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225): tablet 7f4b17b924374b5a91a1727cf13a76eb (table test-table [id=95f63f1d3cde455fb165250e7377db89]) successfully deleted
Sep 23 20:53:30 dist-test-slave-plcg krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:53:30 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:30.784622 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:30.767425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.0.97.62:33405)
0923 20:53:30.768857 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:30.768862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:30.768884 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:53:30.768936 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:30.768940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:30.769010 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:30.769099 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:30.771441 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:30.772224 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:30.774099 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:30.774492 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:30.774603 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:30.778979 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:30.778984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:30.778988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:30.779028 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:30.781882 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:30.782632 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:30.782643 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:30.782645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:30.782773 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:30.783149 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:30.783154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:30.783156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:30.783447 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:30.783700 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:30.784027 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 20:53:30.784271 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":1102,"threads_started":1}
I20260923 20:53:30.788475 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:30.789181 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:30.795977 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0d5a44c13448b88475bc6eac38519a (DEFAULT_TABLE table=test-table [id=99cb85e10de94d83a5f7ae9f39eddd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:30.796062 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0d5a44c13448b88475bc6eac38519a (DEFAULT_TABLE table=test-table [id=99cb85e10de94d83a5f7ae9f39eddd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:30.796465 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0d5a44c13448b88475bc6eac38519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:30.796465 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0d5a44c13448b88475bc6eac38519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:30.796507 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0d5a44c13448b88475bc6eac38519a (DEFAULT_TABLE table=test-table [id=99cb85e10de94d83a5f7ae9f39eddd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:30.796933 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0d5a44c13448b88475bc6eac38519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:30.803014 2742 tablet_bootstrap.cc:492] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: Bootstrap starting.
I20260923 20:53:30.803869 2743 tablet_bootstrap.cc:492] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: Bootstrap starting.
I20260923 20:53:30.804484 2744 tablet_bootstrap.cc:492] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: Bootstrap starting.
I20260923 20:53:30.805145 2742 tablet_bootstrap.cc:654] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:30.806829 2744 tablet_bootstrap.cc:654] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:30.807701 2743 tablet_bootstrap.cc:654] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:30.808038 2742 tablet_bootstrap.cc:492] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: No bootstrap required, opened a new log
I20260923 20:53:30.808221 2742 ts_tablet_manager.cc:1397] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260923 20:53:30.809052 2744 tablet_bootstrap.cc:492] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: No bootstrap required, opened a new log
I20260923 20:53:30.809214 2744 ts_tablet_manager.cc:1397] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260923 20:53:30.809074 2742 raft_consensus.cc:348] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:30.809327 2742 raft_consensus.cc:374] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:30.809434 2742 raft_consensus.cc:729] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86316bb9b7684f31b1653972a98a80f1, State: Initialized, Role: FOLLOWER
I20260923 20:53:30.809684 2742 consensus_queue.cc:260] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:30.810079 2744 raft_consensus.cc:348] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:30.810400 2744 raft_consensus.cc:374] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:30.810514 2744 raft_consensus.cc:729] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924b89b49cba4646a8f869620992fbd0, State: Initialized, Role: FOLLOWER
I20260923 20:53:30.810767 2742 ts_tablet_manager.cc:1428] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 20:53:30.810793 2744 consensus_queue.cc:260] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:30.811381 2744 ts_tablet_manager.cc:1428] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:53:30.811782 2743 tablet_bootstrap.cc:492] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: No bootstrap required, opened a new log
I20260923 20:53:30.811952 2743 ts_tablet_manager.cc:1397] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260923 20:53:30.812515 2743 raft_consensus.cc:348] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:30.812726 2743 raft_consensus.cc:374] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:30.812822 2743 raft_consensus.cc:729] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Initialized, Role: FOLLOWER
I20260923 20:53:30.813055 2743 consensus_queue.cc:260] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:30.814397 2743 ts_tablet_manager.cc:1428] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 20:53:31.182670 2749 raft_consensus.cc:482] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:31.182925 2749 raft_consensus.cc:504] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.183681 2749 leader_election.cc:288] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:31.184376 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d5a44c13448b88475bc6eac38519a" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924b89b49cba4646a8f869620992fbd0" is_pre_election: true
I20260923 20:53:31.184352 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d5a44c13448b88475bc6eac38519a" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1" is_pre_election: true
I20260923 20:53:31.184623 2582 raft_consensus.cc:2454] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 0.
I20260923 20:53:31.184657 2305 raft_consensus.cc:2454] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 0.
I20260923 20:53:31.185170 2372 leader_election.cc:302] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:31.185446 2749 raft_consensus.cc:2781] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:31.185585 2749 raft_consensus.cc:482] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:31.185698 2749 raft_consensus.cc:3037] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:31.187789 2749 raft_consensus.cc:504] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.188468 2749 leader_election.cc:288] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 election: Requested vote from peers 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:31.188983 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d5a44c13448b88475bc6eac38519a" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924b89b49cba4646a8f869620992fbd0"
I20260923 20:53:31.189152 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d5a44c13448b88475bc6eac38519a" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1"
I20260923 20:53:31.189239 2305 raft_consensus.cc:3037] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:31.189373 2582 raft_consensus.cc:3037] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:31.192564 2305 raft_consensus.cc:2454] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 1.
I20260923 20:53:31.192557 2582 raft_consensus.cc:2454] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 1.
I20260923 20:53:31.193082 2372 leader_election.cc:302] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:31.193405 2749 raft_consensus.cc:2781] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:31.194468 2749 raft_consensus.cc:686] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 1 LEADER]: Becoming Leader. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Running, Role: LEADER
I20260923 20:53:31.194844 2749 consensus_queue.cc:237] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.198221 2143 catalog_manager.cc:5495] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "33b26ffc3320426daed3177d6b7888c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:31.274785 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:31.269634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.0.97.2:40769)
0923 20:53:31.270267 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:31.270271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:31.270287 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:53:31.270338 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:31.270341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:31.270412 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:31.270516 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:31.270951 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:31.271673 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:31.272976 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:31.273402 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:31.273538 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:31.274006 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:31.274118 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:31.274221 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:31.274361 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 20:53:31.274438 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":348,"threads_started":1}
I20260923 20:53:31.284468 2305 raft_consensus.cc:1261] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Refusing update from remote peer 33b26ffc3320426daed3177d6b7888c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:31.284512 2582 raft_consensus.cc:1261] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Refusing update from remote peer 33b26ffc3320426daed3177d6b7888c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:31.285063 2749 consensus_queue.cc:1035] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:31.285373 2750 consensus_queue.cc:1035] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:31.347059 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:31.347352 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:31.350639 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Sending DeleteTablet for 3 replicas of tablet 9f0d5a44c13448b88475bc6eac38519a
I20260923 20:53:31.351511 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0d5a44c13448b88475bc6eac38519a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33154
I20260923 20:53:31.351655 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0d5a44c13448b88475bc6eac38519a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37792
I20260923 20:53:31.351660 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0d5a44c13448b88475bc6eac38519a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56720
I20260923 20:53:31.353598 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:31.354974 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:31.357158 2769 tablet_replica.cc:330] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: stopping tablet replica
I20260923 20:53:31.357456 2769 raft_consensus.cc:2229] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:31.357780 2769 raft_consensus.cc:2258] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:31.359040 2769 ts_tablet_manager.cc:1905] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:31.361511 2767 tablet_replica.cc:330] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: stopping tablet replica
I20260923 20:53:31.362003 2767 raft_consensus.cc:2229] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:31.362378 2767 raft_consensus.cc:2258] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:31.363826 2767 ts_tablet_manager.cc:1905] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:31.364239 2768 tablet_replica.cc:330] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: stopping tablet replica
I20260923 20:53:31.364835 2768 raft_consensus.cc:2229] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:31.365258 2768 raft_consensus.cc:2258] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:31.366358 2768 ts_tablet_manager.cc:1905] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:31.366621 2769 ts_tablet_manager.cc:1918] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:31.366781 2769 log.cc:1196] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/9f0d5a44c13448b88475bc6eac38519a
I20260923 20:53:31.367484 2769 ts_tablet_manager.cc:1939] T 9f0d5a44c13448b88475bc6eac38519a P 86316bb9b7684f31b1653972a98a80f1: Deleting consensus metadata
I20260923 20:53:31.368458 2280 tablet_service.cc:1467] Processing CreateTablet for tablet a589feae5a0a459da51b01a043446986 (DEFAULT_TABLE table=test-table [id=7865f91e33614a51a806f48ebab442ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:31.368770 2128 catalog_manager.cc:4841] TS 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171): tablet 9f0d5a44c13448b88475bc6eac38519a (table test-table [id=99cb85e10de94d83a5f7ae9f39eddd49]) successfully deleted
I20260923 20:53:31.368909 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a589feae5a0a459da51b01a043446986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:31.369330 2557 tablet_service.cc:1467] Processing CreateTablet for tablet a589feae5a0a459da51b01a043446986 (DEFAULT_TABLE table=test-table [id=7865f91e33614a51a806f48ebab442ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:31.369805 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a589feae5a0a459da51b01a043446986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:31.370841 2419 tablet_service.cc:1467] Processing CreateTablet for tablet a589feae5a0a459da51b01a043446986 (DEFAULT_TABLE table=test-table [id=7865f91e33614a51a806f48ebab442ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:31.371331 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a589feae5a0a459da51b01a043446986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:31.374267 2768 ts_tablet_manager.cc:1918] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:31.374507 2768 log.cc:1196] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/9f0d5a44c13448b88475bc6eac38519a
I20260923 20:53:31.375038 2768 ts_tablet_manager.cc:1939] T 9f0d5a44c13448b88475bc6eac38519a P 33b26ffc3320426daed3177d6b7888c3: Deleting consensus metadata
I20260923 20:53:31.376267 2129 catalog_manager.cc:4841] TS 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769): tablet 9f0d5a44c13448b88475bc6eac38519a (table test-table [id=99cb85e10de94d83a5f7ae9f39eddd49]) successfully deleted
I20260923 20:53:31.377142 2771 tablet_bootstrap.cc:492] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: Bootstrap starting.
I20260923 20:53:31.379742 2771 tablet_bootstrap.cc:654] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:31.380539 2767 ts_tablet_manager.cc:1918] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:53:31.380785 2767 log.cc:1196] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/9f0d5a44c13448b88475bc6eac38519a
I20260923 20:53:31.381763 2770 tablet_bootstrap.cc:492] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0: Bootstrap starting.
I20260923 20:53:31.382421 2767 ts_tablet_manager.cc:1939] T 9f0d5a44c13448b88475bc6eac38519a P 924b89b49cba4646a8f869620992fbd0: Deleting consensus metadata
I20260923 20:53:31.384143 2129 catalog_manager.cc:4841] TS 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225): tablet 9f0d5a44c13448b88475bc6eac38519a (table test-table [id=99cb85e10de94d83a5f7ae9f39eddd49]) successfully deleted
I20260923 20:53:31.384634 2770 tablet_bootstrap.cc:654] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:31.386566 2773 tablet_bootstrap.cc:492] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: Bootstrap starting.
I20260923 20:53:31.387838 2771 tablet_bootstrap.cc:492] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: No bootstrap required, opened a new log
I20260923 20:53:31.388413 2771 ts_tablet_manager.cc:1397] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260923 20:53:31.388460 2770 tablet_bootstrap.cc:492] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0: No bootstrap required, opened a new log
I20260923 20:53:31.388592 2770 ts_tablet_manager.cc:1397] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260923 20:53:31.388831 2773 tablet_bootstrap.cc:654] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:31.389303 2770 raft_consensus.cc:348] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.389294 2771 raft_consensus.cc:348] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.389550 2770 raft_consensus.cc:374] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:31.389549 2771 raft_consensus.cc:374] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:31.389645 2771 raft_consensus.cc:729] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86316bb9b7684f31b1653972a98a80f1, State: Initialized, Role: FOLLOWER
I20260923 20:53:31.389674 2770 raft_consensus.cc:729] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924b89b49cba4646a8f869620992fbd0, State: Initialized, Role: FOLLOWER
I20260923 20:53:31.389977 2770 consensus_queue.cc:260] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.389977 2771 consensus_queue.cc:260] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.390651 2770 ts_tablet_manager.cc:1428] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 20:53:31.391681 2771 ts_tablet_manager.cc:1428] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 20:53:31.393769 2773 tablet_bootstrap.cc:492] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: No bootstrap required, opened a new log
I20260923 20:53:31.393996 2773 ts_tablet_manager.cc:1397] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:53:31.395068 2773 raft_consensus.cc:348] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.395397 2773 raft_consensus.cc:374] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:31.395550 2773 raft_consensus.cc:729] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Initialized, Role: FOLLOWER
I20260923 20:53:31.395841 2773 consensus_queue.cc:260] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.396373 2773 ts_tablet_manager.cc:1428] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:53:31.677242 2765 raft_consensus.cc:482] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:31.677444 2765 raft_consensus.cc:504] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.678123 2765 leader_election.cc:288] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:31.678722 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a589feae5a0a459da51b01a043446986" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924b89b49cba4646a8f869620992fbd0" is_pre_election: true
I20260923 20:53:31.678745 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a589feae5a0a459da51b01a043446986" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1" is_pre_election: true
I20260923 20:53:31.678956 2305 raft_consensus.cc:2454] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 0.
I20260923 20:53:31.678973 2582 raft_consensus.cc:2454] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 0.
I20260923 20:53:31.679425 2372 leader_election.cc:302] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 86316bb9b7684f31b1653972a98a80f1, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:31.679720 2765 raft_consensus.cc:2781] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:31.679824 2765 raft_consensus.cc:482] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:31.679888 2765 raft_consensus.cc:3037] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:31.682457 2765 raft_consensus.cc:504] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.682996 2765 leader_election.cc:288] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 election: Requested vote from peers 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225), 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171)
I20260923 20:53:31.683611 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a589feae5a0a459da51b01a043446986" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "924b89b49cba4646a8f869620992fbd0"
I20260923 20:53:31.683611 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a589feae5a0a459da51b01a043446986" candidate_uuid: "33b26ffc3320426daed3177d6b7888c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86316bb9b7684f31b1653972a98a80f1"
I20260923 20:53:31.683827 2305 raft_consensus.cc:3037] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:31.683832 2582 raft_consensus.cc:3037] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:31.685933 2305 raft_consensus.cc:2454] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 1.
I20260923 20:53:31.686012 2582 raft_consensus.cc:2454] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33b26ffc3320426daed3177d6b7888c3 in term 1.
I20260923 20:53:31.686498 2372 leader_election.cc:302] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33b26ffc3320426daed3177d6b7888c3, 86316bb9b7684f31b1653972a98a80f1, 924b89b49cba4646a8f869620992fbd0; no voters:
I20260923 20:53:31.686873 2765 raft_consensus.cc:2781] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:31.687055 2765 raft_consensus.cc:686] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 1 LEADER]: Becoming Leader. State: Replica: 33b26ffc3320426daed3177d6b7888c3, State: Running, Role: LEADER
I20260923 20:53:31.687333 2765 consensus_queue.cc:237] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } }
I20260923 20:53:31.690330 2143 catalog_manager.cc:5495] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "33b26ffc3320426daed3177d6b7888c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33b26ffc3320426daed3177d6b7888c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:31.759860 2582 raft_consensus.cc:1261] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Refusing update from remote peer 33b26ffc3320426daed3177d6b7888c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:31.759941 2305 raft_consensus.cc:1261] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Refusing update from remote peer 33b26ffc3320426daed3177d6b7888c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:31.760548 2765 consensus_queue.cc:1035] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86316bb9b7684f31b1653972a98a80f1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:31.760905 2750 consensus_queue.cc:1035] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "924b89b49cba4646a8f869620992fbd0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:53:31.790318 2755 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:53:33.382652 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:33.382895 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:33.387537 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0db54a08269841899d294dee13d0063b: Sending DeleteTablet for 3 replicas of tablet a589feae5a0a459da51b01a043446986
I20260923 20:53:33.389039 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet a589feae5a0a459da51b01a043446986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33154
I20260923 20:53:33.389051 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet a589feae5a0a459da51b01a043446986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37792
I20260923 20:53:33.389099 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet a589feae5a0a459da51b01a043446986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56720
I20260923 20:53:33.391985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2214
I20260923 20:53:33.393229 2802 tablet_replica.cc:330] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: stopping tablet replica
I20260923 20:53:33.393875 2803 tablet_replica.cc:330] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: stopping tablet replica
I20260923 20:53:33.394282 2802 raft_consensus.cc:2229] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:33.394613 2803 raft_consensus.cc:2229] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:33.394865 2801 tablet_replica.cc:330] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0: stopping tablet replica
I20260923 20:53:33.394943 2803 raft_consensus.cc:2258] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:33.395118 2801 raft_consensus.cc:2229] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:33.395412 2801 raft_consensus.cc:2258] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:33.395440 2802 raft_consensus.cc:2258] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:33.396155 2803 ts_tablet_manager.cc:1905] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:33.396373 2801 ts_tablet_manager.cc:1905] T a589feae5a0a459da51b01a043446986 P 924b89b49cba4646a8f869620992fbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:33.396435 2802 ts_tablet_manager.cc:1905] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:53:33 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:33.416915 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:33.393648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.0.97.1:34225)
0923 20:53:33.395070 (+ 1422us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:33.395075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:33.395106 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:53:33.395175 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:33.395180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:33.395264 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:33.395387 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:33.397751 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:33.398958 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:33.400495 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:33.401101 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:33.401221 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:33.411082 (+ 9861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:33.411102 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:33.411119 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:33.411163 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:33.414286 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:33.415065 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:33.415073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:33.415075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:33.415247 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:33.415607 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:33.415611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:33.415613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:33.415961 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:33.416223 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:33.416441 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 20:53:33.416636 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":1109,"threads_started":1}
W20260923 20:53:33.417641 2345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33174
W20260923 20:53:33.418416 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:33.418661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 2214
I20260923 20:53:33.437095 2802 ts_tablet_manager.cc:1918] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:53:33.437120 2803 ts_tablet_manager.cc:1918] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:53:33.437294 2803 log.cc:1196] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/a589feae5a0a459da51b01a043446986
I20260923 20:53:33.437350 2802 log.cc:1196] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/a589feae5a0a459da51b01a043446986
I20260923 20:53:33.437904 2802 ts_tablet_manager.cc:1939] T a589feae5a0a459da51b01a043446986 P 33b26ffc3320426daed3177d6b7888c3: Deleting consensus metadata
I20260923 20:53:33.437934 2803 ts_tablet_manager.cc:1939] T a589feae5a0a459da51b01a043446986 P 86316bb9b7684f31b1653972a98a80f1: Deleting consensus metadata
I20260923 20:53:33.439266 2128 catalog_manager.cc:4841] TS 86316bb9b7684f31b1653972a98a80f1 (127.0.97.3:36171): tablet a589feae5a0a459da51b01a043446986 (table test-table [id=7865f91e33614a51a806f48ebab442ef]) successfully deleted
I20260923 20:53:33.439373 2129 catalog_manager.cc:4841] TS 33b26ffc3320426daed3177d6b7888c3 (127.0.97.2:40769): tablet a589feae5a0a459da51b01a043446986 (table test-table [id=7865f91e33614a51a806f48ebab442ef]) successfully deleted
I20260923 20:53:33.463248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2354
W20260923 20:53:33.463464 2129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34225 (error 108)
I20260923 20:53:33.465709 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:33.464868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.0.1:33180)
0923 20:53:33.465368 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:33.465488 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":326,"threads_started":1}
W20260923 20:53:33.466032 2129 catalog_manager.cc:4584] TS 924b89b49cba4646a8f869620992fbd0 (127.0.97.1:34225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a589feae5a0a459da51b01a043446986: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34225: connect: Connection refused (error 111)
Sep 23 20:53:33 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:53:33.479158 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:33.464045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.0.97.2:40769)
0923 20:53:33.464717 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:33.464724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:33.464749 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:53:33.464803 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:33.464807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:33.464870 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:33.464960 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:33.466517 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:33.467266 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:33.468724 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:33.469220 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:33.469314 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:33.473695 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:33.473709 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:33.473727 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:33.473764 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:33.476512 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:33.477269 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:33.477276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:33.477278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:33.477412 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:33.477741 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:33.477744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:33.477746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:33.478143 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:33.478361 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:33.478607 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 20:53:33.478782 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":395,"threads_started":1}
W20260923 20:53:33.479914 2484 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37822
W20260923 20:53:33.480587 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:33.480721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 2354
I20260923 20:53:33.504240 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2493
Sep 23 20:53:33 dist-test-slave-plcg krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:53:33.520174 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:33.519624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.0.1:33182)
0923 20:53:33.519865 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:33.519974 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 20:53:33.521377 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:33.520932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.0.1:33192)
0923 20:53:33.521081 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:33.521200 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:53:33.522374 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:33.505004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.97.3:36171)
0923 20:53:33.505586 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:33.505590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:33.505607 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:53:33.505662 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:33.505665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:33.505724 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:33.505871 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:33.507238 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:33.508042 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:33.509388 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:33.509985 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:33.510117 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:33.515548 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:33.515564 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:33.515592 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:33.515645 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:33.518810 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:33.519833 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:33.519843 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:33.519846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:33.520038 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:33.520557 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:33.520560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:33.520563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:33.521048 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:33.521295 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:33.521739 (+ 444us) server_negotiation.cc:299] Negotiation successful
0923 20:53:33.522004 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":300,"threads_started":1}
W20260923 20:53:33.523015 2623 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56736
W20260923 20:53:33.523701 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:33.523926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 2493
I20260923 20:53:33.543286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2111
I20260923 20:53:33.544540 2183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:33.660308 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:33.659784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34225 (local address 127.0.0.1:33208)
0923 20:53:33.660005 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:33.660116 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:53:33.660742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 2111
2026-09-23T20:53:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9321 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:35 dist-test-slave-plcg krb5kdc[2813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:35 dist-test-slave-plcg krb5kdc[2813](info): set up 2 sockets
Sep 23 20:53:35 dist-test-slave-plcg krb5kdc[2813](info): commencing operation
krb5kdc: starting...
W20260923 20:53:36.074285 388 mini_kdc.cc:121] Time spent starting KDC: real 2.366s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:36 dist-test-slave-plcg krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:36.273722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39231
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:36.611872 2826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:36.612202 2826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:36.612294 2826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:36.619658 2826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:36.619791 2826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:36.619843 2826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:36.619891 2826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:36.619927 2826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:36.630023 2826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:36.631361 2826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:36.632673 2826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:36.645473 2831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:36.645546 2832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:36.645491 2834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:36.646590 2826 server_base.cc:1034] running on GCE node
I20260923 20:53:36.647511 2826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:36.649111 2826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:36.650352 2826 hybrid_clock.cc:648] HybridClock initialized: now 1790196816650289 us; error 66 us; skew 500 ppm
I20260923 20:53:36.650735 2826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:36 dist-test-slave-plcg krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:36.654181 2826 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:36.655813 2826 webserver.cc:460] Webserver started at http://127.0.97.62:37207/ using document root <none> and password file <none>
I20260923 20:53:36.656579 2826 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:36.656716 2826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:36.657069 2826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:36.660243 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "d26d20d3a84e405795bcf99db8523ea0"
format_stamp: "Formatted at 2026-09-23 20:53:36 on dist-test-slave-plcg"
server_key: "2b5728dedd1f01d32d26d8e4803214c5"
server_key_iv: "619b6af2c63de3d2159d2f995f3b24a9"
server_key_version: "encryptionkey@0"
I20260923 20:53:36.661126 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "d26d20d3a84e405795bcf99db8523ea0"
format_stamp: "Formatted at 2026-09-23 20:53:36 on dist-test-slave-plcg"
server_key: "2b5728dedd1f01d32d26d8e4803214c5"
server_key_iv: "619b6af2c63de3d2159d2f995f3b24a9"
server_key_version: "encryptionkey@0"
I20260923 20:53:36.666935 2826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:53:36.671587 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:36.673285 2826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260923 20:53:36.673519 2826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "d26d20d3a84e405795bcf99db8523ea0"
format_stamp: "Formatted at 2026-09-23 20:53:36 on dist-test-slave-plcg"
server_key: "2b5728dedd1f01d32d26d8e4803214c5"
server_key_iv: "619b6af2c63de3d2159d2f995f3b24a9"
server_key_version: "encryptionkey@0"
I20260923 20:53:36.673769 2826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:36.687747 2826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:36.692350 2826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:36.692657 2826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:36.711690 2826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39231
I20260923 20:53:36.711720 2892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39231 every 8 connection(s)
I20260923 20:53:36.713260 2826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:36.717231 2893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:36.723536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 2826
I20260923 20:53:36.723794 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:36.724561 388 external_mini_cluster.cc:1428] Setting key 017d02f4f7352bf9070cf2ceaa183eef
I20260923 20:53:36.729247 2893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0: Bootstrap starting.
I20260923 20:53:36.732523 2893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:36.733623 2893 log.cc:824] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:36.736653 2893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0: No bootstrap required, opened a new log
Sep 23 20:53:36 dist-test-slave-plcg krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:36.745335 2893 raft_consensus.cc:348] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26d20d3a84e405795bcf99db8523ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39231 } }
I20260923 20:53:36.745769 2893 raft_consensus.cc:374] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:36.745900 2893 raft_consensus.cc:729] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26d20d3a84e405795bcf99db8523ea0, State: Initialized, Role: FOLLOWER
I20260923 20:53:36.746634 2893 consensus_queue.cc:260] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26d20d3a84e405795bcf99db8523ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39231 } }
I20260923 20:53:36.746942 2893 raft_consensus.cc:388] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:36.747099 2893 raft_consensus.cc:482] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:36.747304 2893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:36.749958 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:36.727021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.0.97.62:39231)
0923 20:53:36.727852 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:36.727865 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:36.727912 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:53:36.729903 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:36.729930 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:36.730367 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:36.730833 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:36.732497 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:36.734052 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:36.735467 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:36.736192 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:36.736400 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:36.741002 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:36.741016 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:36.741035 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:36.741094 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:36.744673 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:36.745547 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:36.745557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:36.745565 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:36.745860 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:36.746363 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:36.746368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:36.746372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:36.747181 (+ 809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:36.747506 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:36.747963 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 20:53:36.748471 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":361,"threads_started":1}
I20260923 20:53:36.750734 2893 raft_consensus.cc:504] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26d20d3a84e405795bcf99db8523ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39231 } }
I20260923 20:53:36.751359 2893 leader_election.cc:302] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d26d20d3a84e405795bcf99db8523ea0; no voters:
I20260923 20:53:36.752105 2893 leader_election.cc:288] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:36.752298 2898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:36.753448 2898 raft_consensus.cc:686] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [term 1 LEADER]: Becoming Leader. State: Replica: d26d20d3a84e405795bcf99db8523ea0, State: Running, Role: LEADER
I20260923 20:53:36.754186 2898 consensus_queue.cc:237] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26d20d3a84e405795bcf99db8523ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39231 } }
I20260923 20:53:36.754578 2893 sys_catalog.cc:564] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:36.757980 2899 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d26d20d3a84e405795bcf99db8523ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26d20d3a84e405795bcf99db8523ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39231 } } }
I20260923 20:53:36.758471 2899 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:36.759009 2900 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d26d20d3a84e405795bcf99db8523ea0. Latest consensus state: current_term: 1 leader_uuid: "d26d20d3a84e405795bcf99db8523ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26d20d3a84e405795bcf99db8523ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39231 } } }
I20260923 20:53:36.759367 2900 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:36.759565 2905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:36.767107 2905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:36.779156 2905 catalog_manager.cc:1347] Generated new cluster ID: 9c4ba1cda6154e96b9f97b6a707c07e3
I20260923 20:53:36.779294 2905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:36.788630 2905 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:36.790305 2905 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:36.799090 2905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d26d20d3a84e405795bcf99db8523ea0: Generated new TSK 0
I20260923 20:53:36.799996 2905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:36.857398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39231
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:37.220290 2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:37.220598 2919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:37.220705 2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:37.228194 2919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:37.228317 2919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:37.228433 2919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:37.238557 2919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:37.239882 2919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:37.241185 2919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:37.255308 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:37.255390 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:37.255347 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:37.256920 2919 server_base.cc:1034] running on GCE node
I20260923 20:53:37.257609 2919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:37.258760 2919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:37.260011 2919 hybrid_clock.cc:648] HybridClock initialized: now 1790196817259942 us; error 68 us; skew 500 ppm
I20260923 20:53:37.260398 2919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:37 dist-test-slave-plcg krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:37.264158 2919 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:37.265944 2919 webserver.cc:460] Webserver started at http://127.0.97.1:43315/ using document root <none> and password file <none>
I20260923 20:53:37.266716 2919 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:37.266875 2919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:37.267215 2919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:37.270495 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "6008871190d741f8b76669ba2ad023f7"
format_stamp: "Formatted at 2026-09-23 20:53:37 on dist-test-slave-plcg"
server_key: "2be6cf2d6eeb69a4b8ce1e7ab9af9a8b"
server_key_iv: "17b82ae1497ed383403c002632eff20d"
server_key_version: "encryptionkey@0"
I20260923 20:53:37.271360 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6008871190d741f8b76669ba2ad023f7"
format_stamp: "Formatted at 2026-09-23 20:53:37 on dist-test-slave-plcg"
server_key: "2be6cf2d6eeb69a4b8ce1e7ab9af9a8b"
server_key_iv: "17b82ae1497ed383403c002632eff20d"
server_key_version: "encryptionkey@0"
I20260923 20:53:37.277029 2919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:53:37.281535 2934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:37.283231 2919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:53:37.283469 2919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "6008871190d741f8b76669ba2ad023f7"
format_stamp: "Formatted at 2026-09-23 20:53:37 on dist-test-slave-plcg"
server_key: "2be6cf2d6eeb69a4b8ce1e7ab9af9a8b"
server_key_iv: "17b82ae1497ed383403c002632eff20d"
server_key_version: "encryptionkey@0"
I20260923 20:53:37.283738 2919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:37.299103 2919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:37.303921 2919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:37.304211 2919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:37.305207 2919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:37.306725 2919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:37.306805 2919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:37.306893 2919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:37.306941 2919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:37.341568 2919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33077
I20260923 20:53:37.341614 3046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33077 every 8 connection(s)
I20260923 20:53:37.343003 2919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:37.350591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 2919
I20260923 20:53:37.350864 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:37.351486 388 external_mini_cluster.cc:1428] Setting key 01cce50744c1438e92e434509385b0a1
Sep 23 20:53:37 dist-test-slave-plcg krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:37.365556 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:37.345723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44689 (local address 127.0.97.62:39231)
0923 20:53:37.346236 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:37.346241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:37.347241 (+ 1000us) server_negotiation.cc:407] Connection header received
0923 20:53:37.349239 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:37.349246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:37.349347 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:37.349488 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:37.351388 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.352311 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.354097 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.354593 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.354739 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:37.358880 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:37.358889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:37.358897 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:37.358945 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:37.361961 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:37.363097 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:37.363101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:37.363103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:37.363232 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:37.363742 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:37.363745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:37.363747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:37.363999 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:37.364106 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:37.365070 (+ 964us) server_negotiation.cc:299] Negotiation successful
0923 20:53:37.365251 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":298,"threads_started":1}
I20260923 20:53:37.367271 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:37.346248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39231 (local address 127.0.97.1:44689)
0923 20:53:37.347051 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:37.347093 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:37.348905 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:37.349741 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:37.349751 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:37.350451 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:37.351187 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:37.351209 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.352509 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.352514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:37.353930 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:37.353944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.354888 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.354897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:37.355158 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:37.355957 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:37.355992 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:37.358574 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:37.362178 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:37.362192 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:37.362216 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:37.362941 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:37.363341 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:37.363349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:37.363355 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:37.363630 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:37.364133 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:37.364146 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:37.364650 (+ 504us) client_negotiation.cc:769] Sending connection context
0923 20:53:37.364978 (+ 328us) client_negotiation.cc:240] Negotiation successful
0923 20:53:37.365615 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":306,"threads_started":1}
I20260923 20:53:37.369050 3047 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39231
I20260923 20:53:37.369405 3047 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:53:37.370296 3047 heartbeater.cc:511] Master 127.0.97.62:39231 requested a full tablet report, sending...
I20260923 20:53:37.372617 2858 ts_manager.cc:194] Registered new tserver with Master: 6008871190d741f8b76669ba2ad023f7 (127.0.97.1:33077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 20:53:37.374339 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:37.397501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39231
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:37.740096 3053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:37.740420 3053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:37.740573 3053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:37.748212 3053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:37.748342 3053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:37.748512 3053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:37.758623 3053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:37.760070 3053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:37.761556 3053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:37.775306 3059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:37.775471 3058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:37.775444 3061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:37.776430 3053 server_base.cc:1034] running on GCE node
I20260923 20:53:37.777135 3053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:37.778308 3053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:37.779515 3053 hybrid_clock.cc:648] HybridClock initialized: now 1790196817779447 us; error 64 us; skew 500 ppm
I20260923 20:53:37.779858 3053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:37 dist-test-slave-plcg krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:37.783176 3053 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:37.784732 3053 webserver.cc:460] Webserver started at http://127.0.97.2:39425/ using document root <none> and password file <none>
I20260923 20:53:37.785518 3053 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:37.785630 3053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:37.786006 3053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:37.789320 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "de8a57e1ec27433a876cb89eb727806b"
format_stamp: "Formatted at 2026-09-23 20:53:37 on dist-test-slave-plcg"
server_key: "9c3111d58735abf043674ae48f47dca0"
server_key_iv: "5ea5259cdafa8b821b183d9d0229846b"
server_key_version: "encryptionkey@0"
I20260923 20:53:37.790196 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "de8a57e1ec27433a876cb89eb727806b"
format_stamp: "Formatted at 2026-09-23 20:53:37 on dist-test-slave-plcg"
server_key: "9c3111d58735abf043674ae48f47dca0"
server_key_iv: "5ea5259cdafa8b821b183d9d0229846b"
server_key_version: "encryptionkey@0"
I20260923 20:53:37.795919 3053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:37.799750 3068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:37.801229 3053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:53:37.801429 3053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "de8a57e1ec27433a876cb89eb727806b"
format_stamp: "Formatted at 2026-09-23 20:53:37 on dist-test-slave-plcg"
server_key: "9c3111d58735abf043674ae48f47dca0"
server_key_iv: "5ea5259cdafa8b821b183d9d0229846b"
server_key_version: "encryptionkey@0"
I20260923 20:53:37.801635 3053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:37.817096 3053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:37.821447 3053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:37.821762 3053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:37.822829 3053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:37.824185 3053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:37.824265 3053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:37.824394 3053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:37.824450 3053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:37.857713 3053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41991
I20260923 20:53:37.857765 3180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41991 every 8 connection(s)
I20260923 20:53:37.859025 3053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:37.867085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3053
I20260923 20:53:37.867311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:37.867738 388 external_mini_cluster.cc:1428] Setting key b61b3bffad1f81da694d60cea56df68a
Sep 23 20:53:37 dist-test-slave-plcg krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:37.880754 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:37.861632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56359 (local address 127.0.97.62:39231)
0923 20:53:37.861802 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:37.861807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:37.863088 (+ 1281us) server_negotiation.cc:407] Connection header received
0923 20:53:37.864942 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:37.864947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:37.865053 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:37.865162 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:37.866784 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.867599 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.869674 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.870168 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.870304 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:37.874249 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:37.874258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:37.874266 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:37.874318 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:37.877017 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:37.878133 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:37.878137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:37.878139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:37.878276 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:37.878818 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:37.878821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:37.878823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:37.879159 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:37.879271 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:37.880284 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0923 20:53:37.880496 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:53:37.882421 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:37.862181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39231 (local address 127.0.97.2:56359)
0923 20:53:37.862912 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:37.862954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:37.864642 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:37.865397 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:37.865407 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:37.866016 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:37.866589 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:37.866607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.867756 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.867761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:37.869521 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:37.869535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:37.870401 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:37.870408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:37.870651 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:37.871375 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:37.871400 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:37.873945 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:37.877200 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:37.877214 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:37.877237 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:37.877979 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:37.878391 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:37.878399 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:37.878406 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:37.878690 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:37.879291 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:37.879303 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:37.879809 (+ 506us) client_negotiation.cc:769] Sending connection context
0923 20:53:37.880135 (+ 326us) client_negotiation.cc:240] Negotiation successful
0923 20:53:37.880760 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":298,"threads_started":1}
I20260923 20:53:37.884701 3181 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39231
I20260923 20:53:37.885164 3181 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:37.886312 3181 heartbeater.cc:511] Master 127.0.97.62:39231 requested a full tablet report, sending...
I20260923 20:53:37.888237 2858 ts_manager.cc:194] Registered new tserver with Master: de8a57e1ec27433a876cb89eb727806b (127.0.97.2:41991)
I20260923 20:53:37.889539 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56359
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:37.917564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39231
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:38.266601 3186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:38.266966 3186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:38.267143 3186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:38.274677 3186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:38.274863 3186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:38.275049 3186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:38.285368 3186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:38.286780 3186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:38.288215 3186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:38.302222 3191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:38.302207 3192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:38.302212 3194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:38.303066 3186 server_base.cc:1034] running on GCE node
I20260923 20:53:38.303750 3186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:38.304745 3186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:38.305917 3186 hybrid_clock.cc:648] HybridClock initialized: now 1790196818305865 us; error 50 us; skew 500 ppm
I20260923 20:53:38.306252 3186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:38 dist-test-slave-plcg krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:38.309568 3186 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:38.311193 3186 webserver.cc:460] Webserver started at http://127.0.97.3:34479/ using document root <none> and password file <none>
I20260923 20:53:38.311896 3186 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:38.312003 3186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:38.312278 3186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:38.315382 3186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "e0537dcf08244fadb8b3e006c3ec0b3a"
format_stamp: "Formatted at 2026-09-23 20:53:38 on dist-test-slave-plcg"
server_key: "b2c4bf68ef4942ce3058b025ca87857d"
server_key_iv: "271a5ef8a682ac860b6fe5b8b17bf883"
server_key_version: "encryptionkey@0"
I20260923 20:53:38.316304 3186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e0537dcf08244fadb8b3e006c3ec0b3a"
format_stamp: "Formatted at 2026-09-23 20:53:38 on dist-test-slave-plcg"
server_key: "b2c4bf68ef4942ce3058b025ca87857d"
server_key_iv: "271a5ef8a682ac860b6fe5b8b17bf883"
server_key_version: "encryptionkey@0"
I20260923 20:53:38.321899 3186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:53:38.325654 3201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:38.327188 3186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:53:38.327371 3186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "e0537dcf08244fadb8b3e006c3ec0b3a"
format_stamp: "Formatted at 2026-09-23 20:53:38 on dist-test-slave-plcg"
server_key: "b2c4bf68ef4942ce3058b025ca87857d"
server_key_iv: "271a5ef8a682ac860b6fe5b8b17bf883"
server_key_version: "encryptionkey@0"
I20260923 20:53:38.327615 3186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:38.347553 3186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:38.352010 3186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:38.352335 3186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:38.353358 3186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:38.354934 3186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:38.355029 3186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:38.355195 3186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:38.355247 3186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:38.377650 3047 heartbeater.cc:503] Master 127.0.97.62:39231 was elected leader, sending a full tablet report...
I20260923 20:53:38.388772 3186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41111
I20260923 20:53:38.388818 3313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41111 every 8 connection(s)
I20260923 20:53:38.390184 3186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:38.398227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3186
I20260923 20:53:38.398452 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:38.398878 388 external_mini_cluster.cc:1428] Setting key 98ee9542c56368e41a729a0fe0adaf57
Sep 23 20:53:38 dist-test-slave-plcg krb5kdc[2813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:38.418269 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:38.392799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51633 (local address 127.0.97.62:39231)
0923 20:53:38.393298 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:38.393303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:38.394253 (+ 950us) server_negotiation.cc:407] Connection header received
0923 20:53:38.396097 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:38.396102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:38.396196 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:38.396313 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:38.398003 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:38.398974 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:38.401287 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:38.402256 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:38.402383 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:38.406143 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:38.406148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:38.406152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:38.406189 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:38.415268 (+ 9079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:38.416129 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:38.416135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:38.416138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:38.416296 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:38.416715 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:38.416718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:38.416720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:38.416975 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:38.417082 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:38.417749 (+ 667us) server_negotiation.cc:299] Negotiation successful
0923 20:53:38.417986 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":270,"threads_started":1}
I20260923 20:53:38.419261 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:38.393245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39231 (local address 127.0.97.3:51633)
0923 20:53:38.394078 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:38.394125 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:38.395792 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:38.396571 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:38.396580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:38.397159 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:38.397756 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:38.397776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:38.399147 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:38.399158 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:38.401005 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:38.401035 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:38.402514 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:38.402521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:38.402738 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:38.403509 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:38.403568 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:38.405890 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:38.415434 (+ 9544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:38.415444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:38.415461 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:38.415973 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:38.416398 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:38.416403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:38.416405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:38.416607 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:38.417074 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:38.417080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:38.417420 (+ 340us) client_negotiation.cc:769] Sending connection context
0923 20:53:38.417675 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 20:53:38.418150 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":301,"threads_started":1}
I20260923 20:53:38.420742 3314 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39231
I20260923 20:53:38.421015 3314 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:38.421697 3314 heartbeater.cc:511] Master 127.0.97.62:39231 requested a full tablet report, sending...
I20260923 20:53:38.423221 2858 ts_manager.cc:194] Registered new tserver with Master: e0537dcf08244fadb8b3e006c3ec0b3a (127.0.97.3:41111)
I20260923 20:53:38.423990 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51633
I20260923 20:53:38.427378 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:53:38.461385 3316 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:53:38.458702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.0.97.62:39231)
0923 20:53:38.458925 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:38.458933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:38.459375 (+ 442us) server_negotiation.cc:407] Connection header received
0923 20:53:38.459618 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:38.459622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:38.459688 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:38.459800 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:38.461134 (+ 1334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44596: BlockingRecv error: recv got EOF from 127.0.0.1:44596 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:53:38.464569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2919
W20260923 20:53:38.467293 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2919: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33077: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:53:38.467489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 2919
I20260923 20:53:38.483615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3053
W20260923 20:53:38.486424 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3053: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41991: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:53:38.486646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3053
I20260923 20:53:38.502345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3186
W20260923 20:53:38.505406 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3186: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:41111: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:53:38.505558 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3186
I20260923 20:53:38.521363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 2826
I20260923 20:53:38.522655 2888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:38.628458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 2826
2026-09-23T20:53:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4963 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:38 dist-test-slave-plcg krb5kdc[3332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:38 dist-test-slave-plcg krb5kdc[3332](info): set up 2 sockets
Sep 23 20:53:38 dist-test-slave-plcg krb5kdc[3332](info): commencing operation
krb5kdc: starting...
W20260923 20:53:39.722371 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:39 dist-test-slave-plcg krb5kdc[3332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:39.938760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35713
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:40.278868 3345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:40.279224 3345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:40.279397 3345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:40.287174 3345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:40.287298 3345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:40.287389 3345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:40.287498 3345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:40.287588 3345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:40.298936 3345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:40.300421 3345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:40.302532 3345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:40.316605 3350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:40.316711 3351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:40.316634 3353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:40.319602 3345 server_base.cc:1034] running on GCE node
I20260923 20:53:40.321292 3345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:40.323098 3345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:40.324324 3345 hybrid_clock.cc:648] HybridClock initialized: now 1790196820324270 us; error 55 us; skew 500 ppm
I20260923 20:53:40.324760 3345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:40 dist-test-slave-plcg krb5kdc[3332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:40.328251 3345 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:40.330523 3345 webserver.cc:460] Webserver started at http://127.0.97.62:38833/ using document root <none> and password file <none>
I20260923 20:53:40.331346 3345 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:40.331502 3345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:40.331826 3345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:40.335384 3345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "0f56a78e67134d7ca6c60d0097c8bcfe"
format_stamp: "Formatted at 2026-09-23 20:53:40 on dist-test-slave-plcg"
server_key: "5bcc8d136cfb159a7c60b211d6386a1a"
server_key_iv: "02a5910f407973ead9a48968e14e77e6"
server_key_version: "encryptionkey@0"
I20260923 20:53:40.336396 3345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f56a78e67134d7ca6c60d0097c8bcfe"
format_stamp: "Formatted at 2026-09-23 20:53:40 on dist-test-slave-plcg"
server_key: "5bcc8d136cfb159a7c60b211d6386a1a"
server_key_iv: "02a5910f407973ead9a48968e14e77e6"
server_key_version: "encryptionkey@0"
I20260923 20:53:40.342164 3345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:53:40.346194 3360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:40.347949 3345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:53:40.348157 3345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "0f56a78e67134d7ca6c60d0097c8bcfe"
format_stamp: "Formatted at 2026-09-23 20:53:40 on dist-test-slave-plcg"
server_key: "5bcc8d136cfb159a7c60b211d6386a1a"
server_key_iv: "02a5910f407973ead9a48968e14e77e6"
server_key_version: "encryptionkey@0"
I20260923 20:53:40.348408 3345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:40.368918 3345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:40.374938 3345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:40.375278 3345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:40.396066 3345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35713
I20260923 20:53:40.396121 3411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35713 every 8 connection(s)
I20260923 20:53:40.397541 3345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:40.399164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3345
I20260923 20:53:40.399366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:40.400194 388 external_mini_cluster.cc:1428] Setting key 71e6a73946d13fb0564a983bfc124030
I20260923 20:53:40.402653 3412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:53:40 dist-test-slave-plcg krb5kdc[3332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:40.416986 3412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe: Bootstrap starting.
I20260923 20:53:40.420702 3412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:40.421944 3412 log.cc:824] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:40.425212 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:40.402213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.0.97.62:35713)
0923 20:53:40.403166 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:40.403184 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:40.403257 (+ 73us) server_negotiation.cc:407] Connection header received
0923 20:53:40.405081 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:40.405109 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:40.405553 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:40.406285 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:40.408102 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:40.409743 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:40.411577 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:40.412315 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:40.412461 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:40.417284 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:40.417293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:40.417314 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:40.417360 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:40.420496 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:40.421357 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:40.421365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:40.421372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:40.421574 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:40.422204 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:40.422210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:40.422213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:40.423037 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:40.423389 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:40.423771 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 20:53:40.424191 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":432,"threads_started":1}
I20260923 20:53:40.425266 3412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe: No bootstrap required, opened a new log
I20260923 20:53:40.429368 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:53:40.435853 3412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35713 } }
I20260923 20:53:40.436391 3412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:40.436503 3412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f56a78e67134d7ca6c60d0097c8bcfe, State: Initialized, Role: FOLLOWER
I20260923 20:53:40.437261 3412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35713 } }
I20260923 20:53:40.437551 3412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:40.437716 3412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:40.437937 3412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:40.440237 3412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35713 } }
I20260923 20:53:40.440825 3412 leader_election.cc:302] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f56a78e67134d7ca6c60d0097c8bcfe; no voters:
I20260923 20:53:40.441516 3412 leader_election.cc:288] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:40.441641 3418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:40.443033 3418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [term 1 LEADER]: Becoming Leader. State: Replica: 0f56a78e67134d7ca6c60d0097c8bcfe, State: Running, Role: LEADER
I20260923 20:53:40.443748 3418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35713 } }
I20260923 20:53:40.444212 3412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:40.446918 3420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f56a78e67134d7ca6c60d0097c8bcfe. Latest consensus state: current_term: 1 leader_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35713 } } }
I20260923 20:53:40.447393 3420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:40.448352 3419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f56a78e67134d7ca6c60d0097c8bcfe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35713 } } }
I20260923 20:53:40.448747 3419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:40.451205 3425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:40.459800 3425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:40.472518 3425 catalog_manager.cc:1347] Generated new cluster ID: 82873b62b8ec44ae94c68d0334d46b23
I20260923 20:53:40.472636 3425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:40.486619 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:40.476752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.0.97.62:35713)
0923 20:53:40.476945 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:40.476952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:40.477581 (+ 629us) server_negotiation.cc:407] Connection header received
0923 20:53:40.477792 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:40.477798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:40.477906 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:40.478033 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:40.479332 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:40.480458 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:40.482499 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:40.483183 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:40.483324 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:40.484676 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:40.484682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:40.484687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:40.485567 (+ 880us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:40.485644 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:40.485674 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:40.485791 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:40.486110 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 20:53:40.486226 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:53:40.491721 3425 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:40.493664 3425 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:40.506598 3425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f56a78e67134d7ca6c60d0097c8bcfe: Generated new TSK 0
I20260923 20:53:40.507814 3425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:53:40.519781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3345
I20260923 20:53:40.521202 3407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:40.612974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3345
2026-09-23T20:53:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1978 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:40 dist-test-slave-plcg krb5kdc[3447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:40 dist-test-slave-plcg krb5kdc[3447](info): set up 2 sockets
Sep 23 20:53:40 dist-test-slave-plcg krb5kdc[3447](info): commencing operation
krb5kdc: starting...
W20260923 20:53:41.698333 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:41 dist-test-slave-plcg krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:41.906381 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40777
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:42.241561 3460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:42.241995 3460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:42.242187 3460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:42.249594 3460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:42.249779 3460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:42.249884 3460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:42.249969 3460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:42.250037 3460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:42.260339 3460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:42.261761 3460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:42.263173 3460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:42.275717 3466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:42.275841 3468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:42.275867 3465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:42.276661 3460 server_base.cc:1034] running on GCE node
I20260923 20:53:42.277550 3460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:42.279188 3460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:42.280404 3460 hybrid_clock.cc:648] HybridClock initialized: now 1790196822280358 us; error 48 us; skew 500 ppm
I20260923 20:53:42.280779 3460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:42 dist-test-slave-plcg krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:42.284042 3460 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:42.285708 3460 webserver.cc:460] Webserver started at http://127.0.97.62:39655/ using document root <none> and password file <none>
I20260923 20:53:42.286487 3460 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:42.286600 3460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:42.286857 3460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:42.290072 3460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "18165c3849a84e9583c4befaf6ab3aad"
format_stamp: "Formatted at 2026-09-23 20:53:42 on dist-test-slave-plcg"
server_key: "9fb09a1ad63cfc9acd91ecd7a11b38fc"
server_key_iv: "f2104aed59eac0f8bad5c7c65ff5d145"
server_key_version: "encryptionkey@0"
I20260923 20:53:42.290890 3460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "18165c3849a84e9583c4befaf6ab3aad"
format_stamp: "Formatted at 2026-09-23 20:53:42 on dist-test-slave-plcg"
server_key: "9fb09a1ad63cfc9acd91ecd7a11b38fc"
server_key_iv: "f2104aed59eac0f8bad5c7c65ff5d145"
server_key_version: "encryptionkey@0"
I20260923 20:53:42.296485 3460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:42.300371 3475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:42.301921 3460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:53:42.302078 3460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "18165c3849a84e9583c4befaf6ab3aad"
format_stamp: "Formatted at 2026-09-23 20:53:42 on dist-test-slave-plcg"
server_key: "9fb09a1ad63cfc9acd91ecd7a11b38fc"
server_key_iv: "f2104aed59eac0f8bad5c7c65ff5d145"
server_key_version: "encryptionkey@0"
I20260923 20:53:42.302281 3460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:42.330109 3460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:42.334826 3460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:42.335099 3460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:42.353711 3460 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40777
I20260923 20:53:42.353741 3526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40777 every 8 connection(s)
I20260923 20:53:42.355263 3460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:42.356407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3460
I20260923 20:53:42.356622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:42.357424 388 external_mini_cluster.cc:1428] Setting key b59ab030fc16d6b0e7bbc6fd8b3112d6
I20260923 20:53:42.360220 3527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:53:42 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:42.374569 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad: Bootstrap starting.
I20260923 20:53:42.378130 3527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:42.379222 3527 log.cc:824] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:42.381415 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:42.359559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.0.97.62:40777)
0923 20:53:42.360424 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:42.360437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:42.360478 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:53:42.362148 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:42.362173 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:42.362568 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:42.362930 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:42.364734 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:42.366289 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:42.368058 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:42.368771 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:42.368908 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:42.373885 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:42.373900 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:42.373918 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:42.373965 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:42.377140 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:42.377957 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:42.377970 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:42.377977 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:42.378169 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:42.378616 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:42.378621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:42.378627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:42.379184 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:42.379499 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:42.379977 (+ 478us) server_negotiation.cc:299] Negotiation successful
0923 20:53:42.380441 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":357,"threads_started":1}
I20260923 20:53:42.382412 3527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad: No bootstrap required, opened a new log
I20260923 20:53:42.390074 3527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18165c3849a84e9583c4befaf6ab3aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40777 } }
I20260923 20:53:42.390508 3527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:42.390617 3527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18165c3849a84e9583c4befaf6ab3aad, State: Initialized, Role: FOLLOWER
I20260923 20:53:42.391291 3527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18165c3849a84e9583c4befaf6ab3aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40777 } }
I20260923 20:53:42.391536 3527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:42.391686 3527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:42.391866 3527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:42.394269 3527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18165c3849a84e9583c4befaf6ab3aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40777 } }
I20260923 20:53:42.394805 3527 leader_election.cc:302] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18165c3849a84e9583c4befaf6ab3aad; no voters:
I20260923 20:53:42.395429 3527 leader_election.cc:288] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:42.395594 3532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:42.396325 3532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [term 1 LEADER]: Becoming Leader. State: Replica: 18165c3849a84e9583c4befaf6ab3aad, State: Running, Role: LEADER
I20260923 20:53:42.396992 3532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18165c3849a84e9583c4befaf6ab3aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40777 } }
I20260923 20:53:42.397640 3527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:42.400413 3534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18165c3849a84e9583c4befaf6ab3aad. Latest consensus state: current_term: 1 leader_uuid: "18165c3849a84e9583c4befaf6ab3aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18165c3849a84e9583c4befaf6ab3aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40777 } } }
I20260923 20:53:42.400811 3534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:42.400504 3533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18165c3849a84e9583c4befaf6ab3aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18165c3849a84e9583c4befaf6ab3aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40777 } } }
I20260923 20:53:42.400933 3533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:42.402021 3541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:42.408067 3541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:42.418591 3541 catalog_manager.cc:1347] Generated new cluster ID: 37dad277b66f431db33c8df332920779
I20260923 20:53:42.418759 3541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:42.445889 3541 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:42.447861 3541 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:42.463462 3541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18165c3849a84e9583c4befaf6ab3aad: Generated new TSK 0
I20260923 20:53:42.464635 3541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:42.509788 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40777
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:42.850960 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:42.851295 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:42.851393 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:42.858933 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:42.859066 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:42.859181 3553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:42.869207 3553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:42.870565 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:42.871992 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:42.886160 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:42.886231 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:42.886231 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:42.887626 3553 server_base.cc:1034] running on GCE node
I20260923 20:53:42.888304 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:42.889281 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:42.890484 3553 hybrid_clock.cc:648] HybridClock initialized: now 1790196822890415 us; error 62 us; skew 500 ppm
I20260923 20:53:42.890796 3553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:42 dist-test-slave-plcg krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:42.894382 3553 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:42.896191 3553 webserver.cc:460] Webserver started at http://127.0.97.1:46483/ using document root <none> and password file <none>
I20260923 20:53:42.897028 3553 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:42.897192 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:42.897557 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:42.901042 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "48cef00f1f6d4987bd91b0526c910734"
format_stamp: "Formatted at 2026-09-23 20:53:42 on dist-test-slave-plcg"
server_key: "bc795747718c88c1d52a3336a117da05"
server_key_iv: "6cbecbe358a3f786f5b9889a333912f4"
server_key_version: "encryptionkey@0"
I20260923 20:53:42.902194 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "48cef00f1f6d4987bd91b0526c910734"
format_stamp: "Formatted at 2026-09-23 20:53:42 on dist-test-slave-plcg"
server_key: "bc795747718c88c1d52a3336a117da05"
server_key_iv: "6cbecbe358a3f786f5b9889a333912f4"
server_key_version: "encryptionkey@0"
I20260923 20:53:42.908476 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:53:42.912685 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:42.914433 3553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:53:42.914649 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "48cef00f1f6d4987bd91b0526c910734"
format_stamp: "Formatted at 2026-09-23 20:53:42 on dist-test-slave-plcg"
server_key: "bc795747718c88c1d52a3336a117da05"
server_key_iv: "6cbecbe358a3f786f5b9889a333912f4"
server_key_version: "encryptionkey@0"
I20260923 20:53:42.914883 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:42.927966 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:42.932333 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:42.932670 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:42.933684 3553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:42.935130 3553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:42.935247 3553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:42.935408 3553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:42.935484 3553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:42.970114 3553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44423
I20260923 20:53:42.970172 3680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44423 every 8 connection(s)
I20260923 20:53:42.971499 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:42.979902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3553
I20260923 20:53:42.980141 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:42.980772 388 external_mini_cluster.cc:1428] Setting key 96537d6d5ba6a2ebff00191c8b3df02f
Sep 23 20:53:42 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:42.994417 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:42.974288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56133 (local address 127.0.97.62:40777)
0923 20:53:42.974779 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:42.974784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:42.975995 (+ 1211us) server_negotiation.cc:407] Connection header received
0923 20:53:42.977924 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:42.977930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:42.978025 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:42.978149 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:42.979743 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:42.980588 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:42.982790 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:42.983389 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:42.983547 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:42.987485 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:42.987495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:42.987503 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:42.987555 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:42.990437 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:42.991607 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:42.991612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:42.991614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:42.991750 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:42.992336 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:42.992342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:42.992344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:42.992665 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:42.992797 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:42.993935 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0923 20:53:42.994135 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":282,"threads_started":1}
I20260923 20:53:42.996016 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:42.974914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40777 (local address 127.0.97.1:56133)
0923 20:53:42.975822 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:42.975870 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:42.977576 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:42.978410 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:42.978420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:42.978964 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:42.979548 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:42.979565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:42.980768 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:42.980772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:42.982595 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:42.982622 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:42.983636 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:42.983642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:42.983868 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:42.984620 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:42.984644 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:42.987183 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:42.990628 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:42.990643 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:42.990676 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:42.991417 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:42.991875 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:42.991882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:42.991889 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:42.992162 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:42.992813 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:42.992825 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:42.993344 (+ 519us) client_negotiation.cc:769] Sending connection context
0923 20:53:42.993685 (+ 341us) client_negotiation.cc:240] Negotiation successful
0923 20:53:42.994366 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":375,"threads_started":1}
I20260923 20:53:42.998014 3681 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40777
I20260923 20:53:42.998381 3681 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:42.999264 3681 heartbeater.cc:511] Master 127.0.97.62:40777 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:53:43.001686 3492 ts_manager.cc:194] Registered new tserver with Master: 48cef00f1f6d4987bd91b0526c910734 (127.0.97.1:44423)
I20260923 20:53:43.003290 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:43.030500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40777
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:43.381337 3687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:43.381688 3687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:43.381923 3687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:43.389551 3687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:43.389742 3687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:43.389955 3687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:43.400353 3687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:43.401916 3687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:43.403493 3687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:43.417930 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:43.417853 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:43.417883 3695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:43.418452 3687 server_base.cc:1034] running on GCE node
I20260923 20:53:43.419373 3687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:43.420485 3687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:43.421723 3687 hybrid_clock.cc:648] HybridClock initialized: now 1790196823421673 us; error 42 us; skew 500 ppm
I20260923 20:53:43.422147 3687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:43 dist-test-slave-plcg krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:43.431845 3687 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:43.433501 3687 webserver.cc:460] Webserver started at http://127.0.97.2:41477/ using document root <none> and password file <none>
I20260923 20:53:43.434340 3687 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:43.434456 3687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:43.434715 3687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:43.437904 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "85b2a2420b4c4e25812ce888a75b9a61"
format_stamp: "Formatted at 2026-09-23 20:53:43 on dist-test-slave-plcg"
server_key: "8d063ce3b64f460b7bb0092a04e31160"
server_key_iv: "c4bd219c607a91a3877c364ef00dde2a"
server_key_version: "encryptionkey@0"
I20260923 20:53:43.438766 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "85b2a2420b4c4e25812ce888a75b9a61"
format_stamp: "Formatted at 2026-09-23 20:53:43 on dist-test-slave-plcg"
server_key: "8d063ce3b64f460b7bb0092a04e31160"
server_key_iv: "c4bd219c607a91a3877c364ef00dde2a"
server_key_version: "encryptionkey@0"
I20260923 20:53:43.444622 3687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:43.448632 3702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:43.450362 3687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:53:43.450559 3687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "85b2a2420b4c4e25812ce888a75b9a61"
format_stamp: "Formatted at 2026-09-23 20:53:43 on dist-test-slave-plcg"
server_key: "8d063ce3b64f460b7bb0092a04e31160"
server_key_iv: "c4bd219c607a91a3877c364ef00dde2a"
server_key_version: "encryptionkey@0"
I20260923 20:53:43.450793 3687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:43.467938 3687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:43.472455 3687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:43.472766 3687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:43.473807 3687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:43.475309 3687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:43.475420 3687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:43.475580 3687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:43.475651 3687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:43.510591 3687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44985
I20260923 20:53:43.510638 3814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44985 every 8 connection(s)
I20260923 20:53:43.512023 3687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:43.520812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3687
I20260923 20:53:43.520987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:43.521451 388 external_mini_cluster.cc:1428] Setting key a72c16c99c656c21519a23002ec93b4a
Sep 23 20:53:43 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:43.535772 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:43.514759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55677 (local address 127.0.97.62:40777)
0923 20:53:43.515267 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:43.515271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:43.516137 (+ 866us) server_negotiation.cc:407] Connection header received
0923 20:53:43.518080 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:43.518085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:43.518175 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:43.518305 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:43.519949 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:43.520682 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:43.522660 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:43.523169 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:43.523291 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:43.527620 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:43.527630 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:43.527639 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:43.527693 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:43.531626 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:43.532668 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:43.532677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:43.532683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:43.532869 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:43.533413 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:43.533421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:43.533428 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:43.533884 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:43.534042 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:43.534962 (+ 920us) server_negotiation.cc:299] Negotiation successful
0923 20:53:43.535272 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":266,"threads_started":1}
I20260923 20:53:43.536559 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:43.515213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40777 (local address 127.0.97.2:55677)
0923 20:53:43.515957 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:43.516003 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:43.517709 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:43.518561 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:43.518573 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:43.519182 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:43.519763 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:43.519780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:43.520860 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:43.520864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:43.522507 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:43.522530 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:43.523426 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:43.523436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:43.523699 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:43.524428 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:43.524451 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:43.527293 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:43.531866 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:43.531876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:43.531895 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:43.532409 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:43.533035 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:43.533039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:43.533041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:43.533268 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:43.534029 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:43.534036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:43.534419 (+ 383us) client_negotiation.cc:769] Sending connection context
0923 20:53:43.534807 (+ 388us) client_negotiation.cc:240] Negotiation successful
0923 20:53:43.535273 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":302,"threads_started":1}
I20260923 20:53:43.538416 3815 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40777
I20260923 20:53:43.538764 3815 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:43.539587 3815 heartbeater.cc:511] Master 127.0.97.62:40777 requested a full tablet report, sending...
I20260923 20:53:43.541345 3492 ts_manager.cc:194] Registered new tserver with Master: 85b2a2420b4c4e25812ce888a75b9a61 (127.0.97.2:44985)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:53:43.542312 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:43.569609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40777
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:43.913214 3821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:43.913512 3821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:43.913610 3821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:43.921202 3821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:43.921330 3821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:43.921442 3821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:43.931854 3821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:43.933153 3821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:43.934512 3821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:43.948365 3829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:43.948340 3827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:43.948791 3826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:43.949352 3821 server_base.cc:1034] running on GCE node
I20260923 20:53:43.950075 3821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:43.951107 3821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:43.952369 3821 hybrid_clock.cc:648] HybridClock initialized: now 1790196823952325 us; error 68 us; skew 500 ppm
I20260923 20:53:43.952714 3821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:43 dist-test-slave-plcg krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:43.956352 3821 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:43.958004 3821 webserver.cc:460] Webserver started at http://127.0.97.3:40651/ using document root <none> and password file <none>
I20260923 20:53:43.958762 3821 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:43.958915 3821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:43.959237 3821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:43.962436 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "1377aca8a2064871a3a5d526d6bc5268"
format_stamp: "Formatted at 2026-09-23 20:53:43 on dist-test-slave-plcg"
server_key: "0618009f13b6be00965504fd3fed1b2b"
server_key_iv: "015f72a0bbb706eb98d1a8016a19740f"
server_key_version: "encryptionkey@0"
I20260923 20:53:43.963323 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1377aca8a2064871a3a5d526d6bc5268"
format_stamp: "Formatted at 2026-09-23 20:53:43 on dist-test-slave-plcg"
server_key: "0618009f13b6be00965504fd3fed1b2b"
server_key_iv: "015f72a0bbb706eb98d1a8016a19740f"
server_key_version: "encryptionkey@0"
I20260923 20:53:43.968954 3821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:53:43.972915 3836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:43.974586 3821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:53:43.974895 3821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "1377aca8a2064871a3a5d526d6bc5268"
format_stamp: "Formatted at 2026-09-23 20:53:43 on dist-test-slave-plcg"
server_key: "0618009f13b6be00965504fd3fed1b2b"
server_key_iv: "015f72a0bbb706eb98d1a8016a19740f"
server_key_version: "encryptionkey@0"
I20260923 20:53:43.975173 3821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:43.990137 3821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:43.994825 3821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:43.995198 3821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:43.996345 3821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:43.998001 3821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:43.998123 3821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:43.998292 3821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:43.998379 3821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:44.007618 3681 heartbeater.cc:503] Master 127.0.97.62:40777 was elected leader, sending a full tablet report...
I20260923 20:53:44.036244 3821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35489
I20260923 20:53:44.036346 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35489 every 8 connection(s)
I20260923 20:53:44.037667 3821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:44.040359 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3821
I20260923 20:53:44.040548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:44.041304 388 external_mini_cluster.cc:1428] Setting key 2c322ab5399c942abc7f2ed715c73101
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:44.059701 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:44.040443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42327 (local address 127.0.97.62:40777)
0923 20:53:44.042548 (+ 2105us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:44.042554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:44.042572 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:53:44.044559 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:44.044565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:44.044654 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:44.044785 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:44.046581 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.047369 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.048981 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.049376 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.049477 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:44.053139 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:44.053145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:44.053148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:44.053192 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:44.056503 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.057547 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.057551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.057554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.057689 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.058164 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.058168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.058170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.058433 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:44.058544 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:44.059241 (+ 697us) server_negotiation.cc:299] Negotiation successful
0923 20:53:44.059422 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1990,"thread_start_us":2207,"threads_started":1}
I20260923 20:53:44.060778 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:44.041029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40777 (local address 127.0.97.3:42327)
0923 20:53:44.042084 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:44.042150 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:44.044202 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:44.045096 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:44.045105 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:44.045669 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:44.046393 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:44.046414 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.047547 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.047552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:44.048830 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:44.048844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.049581 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.049586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:44.049799 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:44.050563 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:44.050586 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:44.052858 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:44.056684 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:44.056692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:44.056731 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:44.057385 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:44.057797 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:44.057800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:44.057803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:44.058053 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:44.058538 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:44.058544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:44.058902 (+ 358us) client_negotiation.cc:769] Sending connection context
0923 20:53:44.059162 (+ 260us) client_negotiation.cc:240] Negotiation successful
0923 20:53:44.059577 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":506,"threads_started":1}
I20260923 20:53:44.062409 3949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40777
I20260923 20:53:44.062762 3949 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:44.063540 3949 heartbeater.cc:511] Master 127.0.97.62:40777 requested a full tablet report, sending...
I20260923 20:53:44.065224 3492 ts_manager.cc:194] Registered new tserver with Master: 1377aca8a2064871a3a5d526d6bc5268 (127.0.97.3:35489)
I20260923 20:53:44.066183 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42327
I20260923 20:53:44.070212 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196824, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196824, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:44.122339 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:44.106194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.0.97.62:40777)
0923 20:53:44.106409 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:44.106417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:44.106899 (+ 482us) server_negotiation.cc:407] Connection header received
0923 20:53:44.107151 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:44.107156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:44.107222 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:44.107326 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:44.108713 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.109463 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.110833 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.111258 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.111397 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:44.116646 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:44.116653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:44.116656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:44.116698 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:44.119694 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.120546 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.120550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.120553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.120702 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.121111 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.121115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.121117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.121433 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:44.121628 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:44.121934 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 20:53:44.122117 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 20:53:44.122923 3492 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45006
I20260923 20:53:44.124498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3553
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196824, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:44.141253 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:44.125571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.0.97.1:44423)
0923 20:53:44.126264 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:44.126271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:44.126297 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:53:44.126354 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:44.126360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:44.126571 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:44.126700 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:44.128018 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.129060 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.130437 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.131027 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.131188 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:44.135420 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:44.135436 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:44.135453 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:44.135498 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:44.138567 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.139360 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.139367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.139369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.139493 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.139894 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.139897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.139899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.140253 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:44.140477 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:44.140733 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 20:53:44.140935 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":363,"threads_started":1}
W20260923 20:53:44.142781 3676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57332
W20260923 20:53:44.143944 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:44.144200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3553
I20260923 20:53:44.160938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3687
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196824, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:53:44.177307 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:44.161959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.0.97.2:44985)
0923 20:53:44.162595 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:44.162603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:44.162631 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:53:44.162699 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:44.162705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:44.162927 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:44.163064 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:44.164308 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.165320 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.166725 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.167321 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.167421 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:44.171556 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:44.171569 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:44.171585 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:44.171620 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:44.174612 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.175313 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.175319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.175322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.175438 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.175811 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.175816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.175818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.176228 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:44.176482 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:44.176720 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 20:53:44.176960 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":358,"threads_started":1}
W20260923 20:53:44.178931 3810 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52408
W20260923 20:53:44.180352 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3687: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:44.180583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3687
I20260923 20:53:44.197249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3821
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196824, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:53:44.214579 3963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:44.198292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.0.97.3:35489)
0923 20:53:44.199004 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:44.199013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:44.199047 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:53:44.199108 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:44.199114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:44.199332 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:44.199501 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:44.200830 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.202077 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.203374 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:44.203969 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:44.204116 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:44.208866 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:44.208881 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:44.208898 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:44.208940 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:44.211866 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.212660 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.212670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.212673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.212818 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:44.213154 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:44.213157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:44.213159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:44.213505 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:44.213740 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:44.214063 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 20:53:44.214290 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":341,"threads_started":1}
W20260923 20:53:44.215938 3944 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42312
W20260923 20:53:44.217134 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3821: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:53:44.217306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3821
I20260923 20:53:44.233341 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3460
I20260923 20:53:44.234658 3522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:44.337124 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3460
2026-09-23T20:53:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3730 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3967](info): set up 2 sockets
Sep 23 20:53:44 dist-test-slave-plcg krb5kdc[3967](info): commencing operation
krb5kdc: starting...
W20260923 20:53:45.458403 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:45 dist-test-slave-plcg krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:45.673971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36493
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:46.039770 3980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:46.040094 3980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:46.040189 3980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:46.047816 3980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:46.047946 3980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:46.047993 3980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:46.048043 3980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:46.048087 3980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:46.058655 3980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:46.060024 3980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:46.061383 3980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:46.074385 3988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:46.074446 3986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:46.074810 3985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:46.075546 3980 server_base.cc:1034] running on GCE node
I20260923 20:53:46.076442 3980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:46.078163 3980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:46.079460 3980 hybrid_clock.cc:648] HybridClock initialized: now 1790196826079398 us; error 76 us; skew 500 ppm
I20260923 20:53:46.079872 3980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:46 dist-test-slave-plcg krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:46.083552 3980 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:46.085564 3980 webserver.cc:460] Webserver started at http://127.0.97.62:43993/ using document root <none> and password file <none>
I20260923 20:53:46.086583 3980 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:46.086783 3980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:46.087162 3980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:46.090862 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "d3a5b807bd3e4b6eabedd3274756adc4"
format_stamp: "Formatted at 2026-09-23 20:53:46 on dist-test-slave-plcg"
server_key: "3b9e5bb05ba8f63456b7a176aab6e378"
server_key_iv: "60844bfd84c9270900623657c0333df8"
server_key_version: "encryptionkey@0"
I20260923 20:53:46.091854 3980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "d3a5b807bd3e4b6eabedd3274756adc4"
format_stamp: "Formatted at 2026-09-23 20:53:46 on dist-test-slave-plcg"
server_key: "3b9e5bb05ba8f63456b7a176aab6e378"
server_key_iv: "60844bfd84c9270900623657c0333df8"
server_key_version: "encryptionkey@0"
I20260923 20:53:46.098299 3980 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:53:46.102453 3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:46.104194 3980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:53:46.104418 3980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "d3a5b807bd3e4b6eabedd3274756adc4"
format_stamp: "Formatted at 2026-09-23 20:53:46 on dist-test-slave-plcg"
server_key: "3b9e5bb05ba8f63456b7a176aab6e378"
server_key_iv: "60844bfd84c9270900623657c0333df8"
server_key_version: "encryptionkey@0"
I20260923 20:53:46.104628 3980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:46.124624 3980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:46.130111 3980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:46.130384 3980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:46.151094 3980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36493
I20260923 20:53:46.151113 4046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36493 every 8 connection(s)
I20260923 20:53:46.152648 3980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:46.155618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 3980
I20260923 20:53:46.155859 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:46.156642 388 external_mini_cluster.cc:1428] Setting key 11b4719a7182dc1e7c9d8b5c809cc952
I20260923 20:53:46.157101 4047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:46.172343 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4: Bootstrap starting.
Sep 23 20:53:46 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:46.176329 4047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:46.177572 4047 log.cc:824] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:46.181003 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4: No bootstrap required, opened a new log
I20260923 20:53:46.181561 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:46.158894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.0.97.62:36493)
0923 20:53:46.159761 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:46.159776 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:46.159819 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:53:46.162034 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:46.162060 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:46.162556 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:46.162948 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:46.164741 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:46.166116 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:46.167887 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:46.168644 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:46.168804 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:46.173741 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:46.173750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:46.173766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:46.173871 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:46.177055 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:46.177950 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:46.177957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:46.177964 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:46.178177 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:46.178632 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:46.178640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:46.178646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:46.179330 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:46.179538 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:46.180041 (+ 503us) server_negotiation.cc:299] Negotiation successful
0923 20:53:46.180429 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":399,"threads_started":1}
I20260923 20:53:46.188825 4047 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36493 } }
I20260923 20:53:46.189268 4047 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:46.189385 4047 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a5b807bd3e4b6eabedd3274756adc4, State: Initialized, Role: FOLLOWER
I20260923 20:53:46.190040 4047 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36493 } }
I20260923 20:53:46.190296 4047 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:46.190439 4047 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:46.190624 4047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:46.192863 4047 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36493 } }
I20260923 20:53:46.193424 4047 leader_election.cc:302] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3a5b807bd3e4b6eabedd3274756adc4; no voters:
I20260923 20:53:46.194137 4047 leader_election.cc:288] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:46.194283 4052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:46.195525 4052 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [term 1 LEADER]: Becoming Leader. State: Replica: d3a5b807bd3e4b6eabedd3274756adc4, State: Running, Role: LEADER
I20260923 20:53:46.196169 4052 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36493 } }
I20260923 20:53:46.196394 4047 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:46.199163 4053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36493 } } }
I20260923 20:53:46.199138 4054 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a5b807bd3e4b6eabedd3274756adc4. Latest consensus state: current_term: 1 leader_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a5b807bd3e4b6eabedd3274756adc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36493 } } }
I20260923 20:53:46.199633 4054 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:46.199633 4053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:46.201913 4058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:46.207628 4058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:46.217749 4058 catalog_manager.cc:1347] Generated new cluster ID: 557fb61783094ff8857a4fc3de77d9fb
I20260923 20:53:46.217960 4058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:46.254459 4058 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:46.256057 4058 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:46.266845 4058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3a5b807bd3e4b6eabedd3274756adc4: Generated new TSK 0
I20260923 20:53:46.268043 4058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:46.321101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36493
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:46.661314 4073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:46.661624 4073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:46.661790 4073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:46.669440 4073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:46.669560 4073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:46.669739 4073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:46.679989 4073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:46.681461 4073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:46.682824 4073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:46.696445 4079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:46.696518 4078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:46.697031 4081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:46.697448 4073 server_base.cc:1034] running on GCE node
I20260923 20:53:46.698235 4073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:46.699281 4073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:46.700497 4073 hybrid_clock.cc:648] HybridClock initialized: now 1790196826700424 us; error 65 us; skew 500 ppm
I20260923 20:53:46.700803 4073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:46 dist-test-slave-plcg krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:46.704169 4073 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:46.705852 4073 webserver.cc:460] Webserver started at http://127.0.97.1:46629/ using document root <none> and password file <none>
I20260923 20:53:46.706609 4073 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:46.706759 4073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:46.707096 4073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:46.710345 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "400d9476c3d447d099e876dcb70df088"
format_stamp: "Formatted at 2026-09-23 20:53:46 on dist-test-slave-plcg"
server_key: "acf0fca47e4b3b54e1ae417e48b48869"
server_key_iv: "ab2dcb21556f08e36e99adef933e3625"
server_key_version: "encryptionkey@0"
I20260923 20:53:46.711246 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "400d9476c3d447d099e876dcb70df088"
format_stamp: "Formatted at 2026-09-23 20:53:46 on dist-test-slave-plcg"
server_key: "acf0fca47e4b3b54e1ae417e48b48869"
server_key_iv: "ab2dcb21556f08e36e99adef933e3625"
server_key_version: "encryptionkey@0"
I20260923 20:53:46.716897 4073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:53:46.720830 4088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:46.722402 4073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:53:46.722626 4073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "400d9476c3d447d099e876dcb70df088"
format_stamp: "Formatted at 2026-09-23 20:53:46 on dist-test-slave-plcg"
server_key: "acf0fca47e4b3b54e1ae417e48b48869"
server_key_iv: "ab2dcb21556f08e36e99adef933e3625"
server_key_version: "encryptionkey@0"
I20260923 20:53:46.722870 4073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:46.737583 4073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:46.742399 4073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:46.742734 4073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:46.743754 4073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:46.745169 4073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:46.745278 4073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:46.745433 4073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:46.745509 4073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:46.780871 4073 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35463
I20260923 20:53:46.780975 4200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35463 every 8 connection(s)
I20260923 20:53:46.782357 4073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:46.791571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 4073
I20260923 20:53:46.791826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:46.792611 388 external_mini_cluster.cc:1428] Setting key 86dad68e5461117ecb846b54629ea243
Sep 23 20:53:46 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:46.806284 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:46.785225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48171 (local address 127.0.97.62:36493)
0923 20:53:46.785697 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:46.785702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:46.786839 (+ 1137us) server_negotiation.cc:407] Connection header received
0923 20:53:46.788741 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:46.788746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:46.788819 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:46.788929 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:46.790732 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:46.791455 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:46.793949 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:46.794679 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:46.794825 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:46.799422 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:46.799428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:46.799431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:46.799467 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:46.802512 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:46.803562 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:46.803567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:46.803570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:46.803710 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:46.804239 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:46.804246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:46.804248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:46.804537 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:46.804709 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:46.805542 (+ 833us) server_negotiation.cc:299] Negotiation successful
0923 20:53:46.805809 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":349,"thread_start_us":253,"threads_started":1}
I20260923 20:53:46.807312 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:46.785713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36493 (local address 127.0.97.1:48171)
0923 20:53:46.786621 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:46.786681 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:46.788420 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:46.789239 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:46.789249 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:46.789876 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:46.790516 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:46.790537 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:46.791677 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:46.791685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:46.793752 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:46.793771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:46.794983 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:46.794991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:46.795290 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:46.796119 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:46.796159 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:46.799137 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:46.802711 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:46.802723 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:46.802751 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:46.803386 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:46.803845 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:46.803851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:46.803855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:46.804091 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:46.804682 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:46.804692 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:46.805099 (+ 407us) client_negotiation.cc:769] Sending connection context
0923 20:53:46.805410 (+ 311us) client_negotiation.cc:240] Negotiation successful
0923 20:53:46.806056 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":331,"threads_started":1}
I20260923 20:53:46.809244 4201 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36493
I20260923 20:53:46.809657 4201 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:46.810689 4201 heartbeater.cc:511] Master 127.0.97.62:36493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:53:46.813242 4012 ts_manager.cc:194] Registered new tserver with Master: 400d9476c3d447d099e876dcb70df088 (127.0.97.1:35463)
I20260923 20:53:46.815058 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:46.842600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36493
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:47.186939 4207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:47.187263 4207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:47.187363 4207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:47.195153 4207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:47.195286 4207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:47.195401 4207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:47.205914 4207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:47.207248 4207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:47.208527 4207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:47.222648 4213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:47.222738 4215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:47.222944 4212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:47.223541 4207 server_base.cc:1034] running on GCE node
I20260923 20:53:47.224435 4207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:47.225564 4207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:47.226807 4207 hybrid_clock.cc:648] HybridClock initialized: now 1790196827226710 us; error 97 us; skew 500 ppm
I20260923 20:53:47.227196 4207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:47 dist-test-slave-plcg krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:47.230540 4207 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:47.232205 4207 webserver.cc:460] Webserver started at http://127.0.97.2:41475/ using document root <none> and password file <none>
I20260923 20:53:47.232944 4207 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:47.233104 4207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:47.233421 4207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:47.236528 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "83aa5947b9304ddbac0a2d92df88644b"
format_stamp: "Formatted at 2026-09-23 20:53:47 on dist-test-slave-plcg"
server_key: "aab9c97df4c3a57d4ffb9d7194ca1fad"
server_key_iv: "70a78a0864878345fc185673a558a709"
server_key_version: "encryptionkey@0"
I20260923 20:53:47.237406 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "83aa5947b9304ddbac0a2d92df88644b"
format_stamp: "Formatted at 2026-09-23 20:53:47 on dist-test-slave-plcg"
server_key: "aab9c97df4c3a57d4ffb9d7194ca1fad"
server_key_iv: "70a78a0864878345fc185673a558a709"
server_key_version: "encryptionkey@0"
I20260923 20:53:47.243149 4207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:53:47.246910 4222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:47.248415 4207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:53:47.248607 4207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "83aa5947b9304ddbac0a2d92df88644b"
format_stamp: "Formatted at 2026-09-23 20:53:47 on dist-test-slave-plcg"
server_key: "aab9c97df4c3a57d4ffb9d7194ca1fad"
server_key_iv: "70a78a0864878345fc185673a558a709"
server_key_version: "encryptionkey@0"
I20260923 20:53:47.248831 4207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:47.266103 4207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:47.270893 4207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:47.271214 4207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:47.272226 4207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:47.273679 4207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:47.273782 4207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:47.274012 4207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:47.274088 4207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:47.308086 4207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45301
I20260923 20:53:47.308190 4334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45301 every 8 connection(s)
I20260923 20:53:47.309508 4207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:47.313220 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 4207
I20260923 20:53:47.313427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:47.314029 388 external_mini_cluster.cc:1428] Setting key 8093e357dee98f5765d1b75bbee03587
Sep 23 20:53:47 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:47.335031 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:47.312342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36189 (local address 127.0.97.62:36493)
0923 20:53:47.312861 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:47.312866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:47.316580 (+ 3714us) server_negotiation.cc:407] Connection header received
0923 20:53:47.316631 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:47.316635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:47.316696 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:47.316781 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:47.319211 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.319941 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.322579 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.323345 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.323500 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:47.327355 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:47.327364 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:47.327372 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:47.327422 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:47.331354 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.332242 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.332248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.332252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.332396 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.332920 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.332926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.332929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.333238 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:47.333397 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:47.334282 (+ 885us) server_negotiation.cc:299] Negotiation successful
0923 20:53:47.334551 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":256,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:53:47.335774 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:47.313098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36493 (local address 127.0.97.2:36189)
0923 20:53:47.314022 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:47.314063 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:47.316268 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:47.317118 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:47.317133 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:47.318064 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:47.318983 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:47.319012 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.320142 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.320150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:47.322400 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:47.322416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.323598 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.323603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:47.323828 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:47.324581 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:47.324606 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:47.327045 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:47.331576 (+ 4531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:47.331586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:47.331603 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:47.332090 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:47.332528 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:47.332533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:47.332538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:47.332791 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:47.333373 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:47.333381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:47.333770 (+ 389us) client_negotiation.cc:769] Sending connection context
0923 20:53:47.334158 (+ 388us) client_negotiation.cc:240] Negotiation successful
0923 20:53:47.334595 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":395,"threads_started":1}
I20260923 20:53:47.337316 4335 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36493
I20260923 20:53:47.337684 4335 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 20:53:47.338654 4335 heartbeater.cc:511] Master 127.0.97.62:36493 requested a full tablet report, sending...
I20260923 20:53:47.340366 4012 ts_manager.cc:194] Registered new tserver with Master: 83aa5947b9304ddbac0a2d92df88644b (127.0.97.2:45301)
I20260923 20:53:47.341229 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:47.362952 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36493
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:47.711210 4341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:47.711584 4341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:47.711769 4341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:47.720011 4341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:47.720209 4341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:47.720383 4341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:47.730794 4341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:47.732206 4341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:47.733616 4341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:47.748344 4346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:47.748409 4349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:47.748438 4347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:47.749181 4341 server_base.cc:1034] running on GCE node
I20260923 20:53:47.749789 4341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:47.750835 4341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:47.752040 4341 hybrid_clock.cc:648] HybridClock initialized: now 1790196827751971 us; error 67 us; skew 500 ppm
I20260923 20:53:47.752329 4341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:47 dist-test-slave-plcg krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:47.755823 4341 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:47.757372 4341 webserver.cc:460] Webserver started at http://127.0.97.3:46423/ using document root <none> and password file <none>
I20260923 20:53:47.758176 4341 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:47.758306 4341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:47.758576 4341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:47.761574 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "bbb96227b8144b87a0eb48bc6affc248"
format_stamp: "Formatted at 2026-09-23 20:53:47 on dist-test-slave-plcg"
server_key: "7a55adc2196a93dec1c5b5d79c5b8d0d"
server_key_iv: "e437e6ea75d2cf3f064145472de58e60"
server_key_version: "encryptionkey@0"
I20260923 20:53:47.762470 4341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bbb96227b8144b87a0eb48bc6affc248"
format_stamp: "Formatted at 2026-09-23 20:53:47 on dist-test-slave-plcg"
server_key: "7a55adc2196a93dec1c5b5d79c5b8d0d"
server_key_iv: "e437e6ea75d2cf3f064145472de58e60"
server_key_version: "encryptionkey@0"
I20260923 20:53:47.768066 4341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:47.772064 4356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:47.773586 4341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:53:47.773804 4341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "bbb96227b8144b87a0eb48bc6affc248"
format_stamp: "Formatted at 2026-09-23 20:53:47 on dist-test-slave-plcg"
server_key: "7a55adc2196a93dec1c5b5d79c5b8d0d"
server_key_iv: "e437e6ea75d2cf3f064145472de58e60"
server_key_version: "encryptionkey@0"
I20260923 20:53:47.774093 4341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:47.800634 4341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:47.805428 4341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:47.805754 4341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:47.806829 4341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:47.808281 4341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:47.808398 4341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:47.808540 4341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:47.808590 4341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:47.818389 4201 heartbeater.cc:503] Master 127.0.97.62:36493 was elected leader, sending a full tablet report...
I20260923 20:53:47.843251 4341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36675
I20260923 20:53:47.843284 4468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36675 every 8 connection(s)
I20260923 20:53:47.844545 4341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:47.853463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 4341
I20260923 20:53:47.853642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:47.854130 388 external_mini_cluster.cc:1428] Setting key 507f87e83340b9f4ebef9ffdb671a727
Sep 23 20:53:47 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:47.866806 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:47.847308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59315 (local address 127.0.97.62:36493)
0923 20:53:47.847796 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:47.847801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:47.848685 (+ 884us) server_negotiation.cc:407] Connection header received
0923 20:53:47.850688 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:47.850694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:47.850818 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:47.850941 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:47.853230 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.854416 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.856309 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.856724 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.856826 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:47.860742 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:47.860749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:47.860753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:47.860792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:47.863785 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.864569 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.864573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.864576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.864704 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.865107 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.865111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.865113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.865402 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:47.865534 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:47.866312 (+ 778us) server_negotiation.cc:299] Negotiation successful
0923 20:53:47.866505 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
I20260923 20:53:47.867789 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:47.847744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36493 (local address 127.0.97.3:59315)
0923 20:53:47.848496 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:47.848561 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:47.850372 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:47.851206 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:47.851216 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:47.851882 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:47.852943 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:47.852992 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.854613 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.854618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:47.856134 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:47.856148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.856978 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.856988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:47.857326 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:47.858108 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:47.858132 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:47.860463 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:47.863966 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:47.863974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:47.863989 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:47.864447 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:47.864808 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:47.864811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:47.864813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:47.864998 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:47.865523 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:47.865529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:47.865948 (+ 419us) client_negotiation.cc:769] Sending connection context
0923 20:53:47.866205 (+ 257us) client_negotiation.cc:240] Negotiation successful
0923 20:53:47.866599 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":299,"threads_started":1}
I20260923 20:53:47.869246 4469 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36493
I20260923 20:53:47.869635 4469 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:47.870442 4469 heartbeater.cc:511] Master 127.0.97.62:36493 requested a full tablet report, sending...
I20260923 20:53:47.871914 4012 ts_manager.cc:194] Registered new tserver with Master: bbb96227b8144b87a0eb48bc6affc248 (127.0.97.3:36675)
I20260923 20:53:47.872736 4012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59315
I20260923 20:53:47.882524 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:53:47 dist-test-slave-plcg krb5kdc[3967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:53:47 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:47.935084 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:47.919546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.0.97.1:35463)
0923 20:53:47.920222 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:47.920230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:47.920258 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:53:47.920319 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:47.920326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:47.920545 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:47.920675 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:47.922341 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.923347 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.924657 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.925220 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.925318 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:47.929392 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:47.929403 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:47.929418 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:47.929457 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:47.932206 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.932960 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.932968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.932970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.933117 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.933541 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.933546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.933548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.934009 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:47.934254 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:47.934551 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 20:53:47.934759 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":375,"threads_started":1}
Sep 23 20:53:47 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:47.953747 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:47.939608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.0.97.62:36493)
0923 20:53:47.939830 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:47.939841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:47.939943 (+ 102us) server_negotiation.cc:407] Connection header received
0923 20:53:47.940070 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:47.940074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:47.940171 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:47.940254 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:47.941486 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.942510 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.943702 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.944103 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.944200 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:47.948198 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:47.948203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:47.948207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:47.948245 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:47.950900 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.951596 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.951600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.951602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.951722 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.952175 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.952182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.952186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.952543 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:47.952817 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:47.953172 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 20:53:47.953415 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 20:53:47.954367 4012 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38024
I20260923 20:53:47.955492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 4073
I20260923 20:53:47.968986 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:47.956497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.97.1:35463)
0923 20:53:47.956725 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:47.956731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:47.956915 (+ 184us) server_negotiation.cc:407] Connection header received
0923 20:53:47.957109 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:47.957114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:47.957186 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:47.957293 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:47.958562 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.959234 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.960485 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:47.960935 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:47.961077 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:47.963463 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:47.963468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:47.963472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:47.963508 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:47.966283 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.966997 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.967010 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.967013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.967137 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:47.967515 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:47.967520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:47.967523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:47.967783 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:47.968035 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:47.968325 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 20:53:47.968547 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:53:47.969790 4196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:48.052596 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 4073
I20260923 20:53:48.069315 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 4207
Sep 23 20:53:48 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:53:48.087033 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:48.070265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.0.97.2:45301)
0923 20:53:48.070840 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:48.070847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:48.070875 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:53:48.070943 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:48.070949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:48.071164 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:48.071300 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:48.072499 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:48.073537 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:48.074973 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:48.075545 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:48.075664 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:48.080142 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:48.080157 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:48.080172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:48.080214 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:48.084177 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:48.085082 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:48.085098 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:48.085101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:48.085254 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:48.085603 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:48.085606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:48.085608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:48.086040 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:48.086290 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:48.086540 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 20:53:48.086741 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":312,"threads_started":1}
I20260923 20:53:48.088317 4330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:48.172289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 4207
I20260923 20:53:48.188865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 4341
Sep 23 20:53:48 dist-test-slave-plcg krb5kdc[3967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:53:48.208807 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:48.190140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.0.97.3:36675)
0923 20:53:48.190899 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:48.190906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:48.190942 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:53:48.191033 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:48.191041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:48.191276 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:48.191429 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:48.193122 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:48.194413 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:48.195873 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:48.196687 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:48.196816 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:48.201473 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:48.201488 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:48.201505 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:48.201551 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:48.204906 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:48.205809 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:48.205937 (+ 128us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:48.205941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:48.206096 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:48.206588 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:48.206592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:48.206594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:48.206991 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:48.207266 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:48.208111 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 20:53:48.208420 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":423,"threads_started":1}
I20260923 20:53:48.210456 4464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:48.295351 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 4341
I20260923 20:53:48.311524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 3980
I20260923 20:53:48.312734 4042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:48.418190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 3980
2026-09-23T20:53:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4081 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:48 dist-test-slave-plcg krb5kdc[4489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:48 dist-test-slave-plcg krb5kdc[4489](info): set up 2 sockets
Sep 23 20:53:48 dist-test-slave-plcg krb5kdc[4489](info): commencing operation
krb5kdc: starting...
W20260923 20:53:49.518420 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:53:49 dist-test-slave-plcg krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:53:49.727195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35323
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:50.071202 4502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:50.071532 4502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:50.071630 4502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:50.079169 4502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:50.079308 4502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:50.079355 4502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:50.079406 4502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:50.079454 4502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:50.089462 4502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:50.090883 4502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:50.092160 4502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:50.104095 4510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:50.104167 4508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:50.104305 4507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:50.104982 4502 server_base.cc:1034] running on GCE node
I20260923 20:53:50.105878 4502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:50.107348 4502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:50.108556 4502 hybrid_clock.cc:648] HybridClock initialized: now 1790196830108515 us; error 39 us; skew 500 ppm
I20260923 20:53:50.108958 4502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:50 dist-test-slave-plcg krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:50.112295 4502 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:53:50.113600 4502 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:50.113724 4502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:50.114111 4502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:50.117241 4502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "be006687ad2244b18accc9a0f68dfe94"
format_stamp: "Formatted at 2026-09-23 20:53:50 on dist-test-slave-plcg"
server_key: "ad8cccc036397b7d74818d5e2e1b1d6b"
server_key_iv: "192fd5e952b2e95022b0a19cec132ecf"
server_key_version: "encryptionkey@0"
I20260923 20:53:50.118191 4502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "be006687ad2244b18accc9a0f68dfe94"
format_stamp: "Formatted at 2026-09-23 20:53:50 on dist-test-slave-plcg"
server_key: "ad8cccc036397b7d74818d5e2e1b1d6b"
server_key_iv: "192fd5e952b2e95022b0a19cec132ecf"
server_key_version: "encryptionkey@0"
I20260923 20:53:50.123752 4502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:53:50.127943 4516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:50.129534 4502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:53:50.129781 4502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "be006687ad2244b18accc9a0f68dfe94"
format_stamp: "Formatted at 2026-09-23 20:53:50 on dist-test-slave-plcg"
server_key: "ad8cccc036397b7d74818d5e2e1b1d6b"
server_key_iv: "192fd5e952b2e95022b0a19cec132ecf"
server_key_version: "encryptionkey@0"
I20260923 20:53:50.130125 4502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:50.149641 4502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:50.154393 4502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:50.154700 4502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:50.174101 4502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35323
I20260923 20:53:50.174115 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35323 every 8 connection(s)
I20260923 20:53:50.175122 4502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:50.177752 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 4502
I20260923 20:53:50.178027 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:50.178653 388 external_mini_cluster.cc:1428] Setting key 87a6e6ea1c1351575eaba77404313741
I20260923 20:53:50.179086 4568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:50.194139 4568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94: Bootstrap starting.
Sep 23 20:53:50 dist-test-slave-plcg krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:50.198213 4568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:50.199680 4568 log.cc:824] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:50.203657 4568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94: No bootstrap required, opened a new log
I20260923 20:53:50.205483 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:50.180680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.0.97.62:35323)
0923 20:53:50.181554 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:50.181570 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:50.181621 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:53:50.184085 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:50.184125 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:50.184734 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:50.185272 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:50.186759 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:50.188725 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:50.190162 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:50.191349 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:50.191563 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:50.195958 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:50.195970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:50.195992 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:50.196069 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:50.200028 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:50.201219 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:50.201229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:50.201238 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:50.201492 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:50.202045 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:50.202049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:50.202052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:50.202813 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:50.203155 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:50.203584 (+ 429us) server_negotiation.cc:299] Negotiation successful
0923 20:53:50.204036 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":377,"threads_started":1}
I20260923 20:53:50.212556 4568 raft_consensus.cc:348] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be006687ad2244b18accc9a0f68dfe94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260923 20:53:50.213107 4568 raft_consensus.cc:374] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:50.213245 4568 raft_consensus.cc:729] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be006687ad2244b18accc9a0f68dfe94, State: Initialized, Role: FOLLOWER
I20260923 20:53:50.213992 4568 consensus_queue.cc:260] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be006687ad2244b18accc9a0f68dfe94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260923 20:53:50.214205 4568 raft_consensus.cc:388] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:50.214306 4568 raft_consensus.cc:482] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:50.214433 4568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:50.216596 4568 raft_consensus.cc:504] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be006687ad2244b18accc9a0f68dfe94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260923 20:53:50.217088 4568 leader_election.cc:302] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be006687ad2244b18accc9a0f68dfe94; no voters:
I20260923 20:53:50.217676 4568 leader_election.cc:288] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:50.217886 4573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:50.218559 4573 raft_consensus.cc:686] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [term 1 LEADER]: Becoming Leader. State: Replica: be006687ad2244b18accc9a0f68dfe94, State: Running, Role: LEADER
I20260923 20:53:50.219084 4573 consensus_queue.cc:237] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be006687ad2244b18accc9a0f68dfe94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } }
I20260923 20:53:50.219838 4568 sys_catalog.cc:564] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:50.222081 4574 sys_catalog.cc:455] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be006687ad2244b18accc9a0f68dfe94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be006687ad2244b18accc9a0f68dfe94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } }
I20260923 20:53:50.222573 4574 sys_catalog.cc:458] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:50.223489 4581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:50.223380 4575 sys_catalog.cc:455] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be006687ad2244b18accc9a0f68dfe94. Latest consensus state: current_term: 1 leader_uuid: "be006687ad2244b18accc9a0f68dfe94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be006687ad2244b18accc9a0f68dfe94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35323 } } }
I20260923 20:53:50.225183 4575 sys_catalog.cc:458] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:50.229287 4581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:50.239403 4581 catalog_manager.cc:1347] Generated new cluster ID: e8add62f0fc741e79855fa3d514ea630
I20260923 20:53:50.239540 4581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:50.266674 4581 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:50.268049 4581 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:50.275931 4581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94: Generated new TSK 0
I20260923 20:53:50.276896 4581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:53:50.341460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:50.691219 4594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:50.691519 4594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:50.691685 4594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:50.699429 4594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:50.699543 4594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:50.699713 4594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:50.709970 4594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:50.711604 4594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:50.712955 4594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:50.726573 4600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:50.726573 4602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:50.727139 4599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:50.727643 4594 server_base.cc:1034] running on GCE node
I20260923 20:53:50.728286 4594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:50.729370 4594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:50.730619 4594 hybrid_clock.cc:648] HybridClock initialized: now 1790196830730576 us; error 52 us; skew 500 ppm
I20260923 20:53:50.731046 4594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:50 dist-test-slave-plcg krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:50.734543 4594 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:53:50.735831 4594 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:50.735963 4594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:50.736312 4594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:50.739802 4594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "354e8b982d0740cfb9daf993abcbaac6"
format_stamp: "Formatted at 2026-09-23 20:53:50 on dist-test-slave-plcg"
server_key: "aad393098dab7c27c9ff35b33bcac61e"
server_key_iv: "2653a0143122b0d1c33bd0c35d702eef"
server_key_version: "encryptionkey@0"
I20260923 20:53:50.740654 4594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "354e8b982d0740cfb9daf993abcbaac6"
format_stamp: "Formatted at 2026-09-23 20:53:50 on dist-test-slave-plcg"
server_key: "aad393098dab7c27c9ff35b33bcac61e"
server_key_iv: "2653a0143122b0d1c33bd0c35d702eef"
server_key_version: "encryptionkey@0"
I20260923 20:53:50.746111 4594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260923 20:53:50.749682 4608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:50.751174 4594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:53:50.751390 4594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "354e8b982d0740cfb9daf993abcbaac6"
format_stamp: "Formatted at 2026-09-23 20:53:50 on dist-test-slave-plcg"
server_key: "aad393098dab7c27c9ff35b33bcac61e"
server_key_iv: "2653a0143122b0d1c33bd0c35d702eef"
server_key_version: "encryptionkey@0"
I20260923 20:53:50.751631 4594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:50.780607 4594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:50.785030 4594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:50.785337 4594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:50.786401 4594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:50.787763 4594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:50.787894 4594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:50.788022 4594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:50.788110 4594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:50.821942 4594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34835
I20260923 20:53:50.821957 4720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34835 every 8 connection(s)
I20260923 20:53:50.822849 4594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:50.826750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 4594
I20260923 20:53:50.826968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:50.827775 388 external_mini_cluster.cc:1428] Setting key 80f9b923a781560de3d51f9911e0ec34
Sep 23 20:53:50 dist-test-slave-plcg krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:50.846652 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:50.825386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49345 (local address 127.0.97.62:35323)
0923 20:53:50.825910 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:50.825915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:50.826994 (+ 1079us) server_negotiation.cc:407] Connection header received
0923 20:53:50.829501 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:50.829509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:50.829619 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:50.829738 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:50.832062 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:50.832768 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:50.835528 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:50.835941 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:50.836130 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:50.840429 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:50.840434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:50.840438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:50.840472 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:50.843040 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:50.843937 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:50.843946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:50.843952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:50.844126 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:50.844581 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:50.844589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:50.844596 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:50.845026 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:50.845204 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:50.845955 (+ 751us) server_negotiation.cc:299] Negotiation successful
0923 20:53:50.846222 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260923 20:53:50.847393 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:50.825921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.1:49345)
0923 20:53:50.826802 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:50.826860 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:50.829158 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:50.830065 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:50.830080 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:50.830899 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:50.831833 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:50.831866 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:50.832952 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:50.832960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:50.835326 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:50.835347 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:50.836231 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:50.836242 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:50.836500 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:50.837325 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:50.837361 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:50.840123 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:50.843222 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:50.843233 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:50.843253 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:50.843724 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:50.844253 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:50.844257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:50.844259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:50.844449 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:50.845181 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:50.845187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:50.845529 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 20:53:50.845768 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 20:53:50.846250 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:53:50.849077 4721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260923 20:53:50.849407 4721 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:50.850126 4721 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 20:53:50.852587 4533 ts_manager.cc:194] Registered new tserver with Master: 354e8b982d0740cfb9daf993abcbaac6 (127.0.97.1:34835)
I20260923 20:53:50.854816 4533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:53:50.875617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:51.211429 4727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:51.211786 4727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:51.211961 4727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:51.219410 4727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:51.219584 4727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:51.219791 4727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:51.229868 4727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:51.231377 4727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:51.232749 4727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:51.246737 4733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:51.246831 4735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:51.246767 4732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:51.248077 4727 server_base.cc:1034] running on GCE node
I20260923 20:53:51.248806 4727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:51.249994 4727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:51.251245 4727 hybrid_clock.cc:648] HybridClock initialized: now 1790196831251199 us; error 47 us; skew 500 ppm
I20260923 20:53:51.251595 4727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:51 dist-test-slave-plcg krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:51.255031 4727 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:53:51.256384 4727 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:51.256526 4727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:51.256868 4727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:51.260030 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0"
format_stamp: "Formatted at 2026-09-23 20:53:51 on dist-test-slave-plcg"
server_key: "6ad7d72fb0c447524e6d20ac37d9eaf9"
server_key_iv: "39cada748d94377ef0c3bf5405a02391"
server_key_version: "encryptionkey@0"
I20260923 20:53:51.260911 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0"
format_stamp: "Formatted at 2026-09-23 20:53:51 on dist-test-slave-plcg"
server_key: "6ad7d72fb0c447524e6d20ac37d9eaf9"
server_key_iv: "39cada748d94377ef0c3bf5405a02391"
server_key_version: "encryptionkey@0"
I20260923 20:53:51.266402 4727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:53:51.270365 4741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:51.271898 4727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:53:51.272109 4727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0"
format_stamp: "Formatted at 2026-09-23 20:53:51 on dist-test-slave-plcg"
server_key: "6ad7d72fb0c447524e6d20ac37d9eaf9"
server_key_iv: "39cada748d94377ef0c3bf5405a02391"
server_key_version: "encryptionkey@0"
I20260923 20:53:51.272360 4727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:51.292598 4727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:51.297230 4727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:51.297576 4727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:51.298674 4727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:51.300206 4727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:51.300374 4727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:51.300541 4727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:51.300642 4727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:51.335481 4727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33779
I20260923 20:53:51.335513 4853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33779 every 8 connection(s)
I20260923 20:53:51.336355 4727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:51.345896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 4727
I20260923 20:53:51.346103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:51.346647 388 external_mini_cluster.cc:1428] Setting key 40fdfd059aee6d7864470a861df3c0d3
Sep 23 20:53:51 dist-test-slave-plcg krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:51.359839 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.338829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37523 (local address 127.0.97.62:35323)
0923 20:53:51.339024 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:51.339028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:51.340359 (+ 1331us) server_negotiation.cc:407] Connection header received
0923 20:53:51.342336 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:51.342342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:51.342442 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:51.342567 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:51.344191 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.344884 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.347273 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.347914 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.348066 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.352587 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:51.352598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:51.352606 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:51.352660 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:51.356185 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:51.357256 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:51.357260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:51.357263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:51.357409 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:51.357959 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:51.357963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:51.357966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:51.358280 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:51.358409 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:51.359371 (+ 962us) server_negotiation.cc:299] Negotiation successful
0923 20:53:51.359573 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:53:51.361354 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.339335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.2:37523)
0923 20:53:51.340089 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:51.340136 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:51.342027 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:51.342804 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:51.342814 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:51.343371 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:51.343978 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.343997 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.345068 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.345072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.347070 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.347088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.348183 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.348193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.348496 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.349280 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:51.349316 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:51.352291 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:51.356364 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:51.356377 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:51.356400 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:51.357102 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:51.357527 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:51.357535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:51.357542 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:51.357801 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:51.358427 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:51.358438 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:51.358930 (+ 492us) client_negotiation.cc:769] Sending connection context
0923 20:53:51.359245 (+ 315us) client_negotiation.cc:240] Negotiation successful
0923 20:53:51.359861 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":301,"threads_started":1}
I20260923 20:53:51.363095 4854 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260923 20:53:51.363442 4854 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:51.364212 4854 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
I20260923 20:53:51.366103 4533 ts_manager.cc:194] Registered new tserver with Master: dd08ba228d6f4f348ae4ffa8a8eb53f0 (127.0.97.2:33779)
I20260923 20:53:51.367313 4533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37523
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:53:51.397224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:53:51.728288 4859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:51.728632 4859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:51.728804 4859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:51.736621 4859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:51.736785 4859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:51.736974 4859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:51.747099 4859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:51.748505 4859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:51.749835 4859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:51.763545 4864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:51.763586 4865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:51.763517 4867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:51.764746 4859 server_base.cc:1034] running on GCE node
I20260923 20:53:51.765448 4859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:51.766564 4859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:51.767764 4859 hybrid_clock.cc:648] HybridClock initialized: now 1790196831767697 us; error 60 us; skew 500 ppm
I20260923 20:53:51.768097 4859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:53:51 dist-test-slave-plcg krb5kdc[4489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:53:51.771518 4859 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:53:51.772845 4859 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:51.772987 4859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:51.773322 4859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:51.776641 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "1dcb67d744614f669a06279fa40243cd"
format_stamp: "Formatted at 2026-09-23 20:53:51 on dist-test-slave-plcg"
server_key: "5d9ef53d03383ec40553bbb80f6c2c82"
server_key_iv: "a15784c5b68e943cfe063341837b788f"
server_key_version: "encryptionkey@0"
I20260923 20:53:51.777537 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1dcb67d744614f669a06279fa40243cd"
format_stamp: "Formatted at 2026-09-23 20:53:51 on dist-test-slave-plcg"
server_key: "5d9ef53d03383ec40553bbb80f6c2c82"
server_key_iv: "a15784c5b68e943cfe063341837b788f"
server_key_version: "encryptionkey@0"
I20260923 20:53:51.783085 4859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260923 20:53:51.786890 4873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:51.788398 4859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:53:51.788595 4859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "1dcb67d744614f669a06279fa40243cd"
format_stamp: "Formatted at 2026-09-23 20:53:51 on dist-test-slave-plcg"
server_key: "5d9ef53d03383ec40553bbb80f6c2c82"
server_key_iv: "a15784c5b68e943cfe063341837b788f"
server_key_version: "encryptionkey@0"
I20260923 20:53:51.788833 4859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:51.802266 4859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:51.807101 4859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:51.807402 4859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:51.808429 4859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:51.809792 4859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:51.809947 4859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:51.810083 4859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:51.810186 4859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:51.844609 4859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39557
I20260923 20:53:51.844662 4985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39557 every 8 connection(s)
I20260923 20:53:51.845527 4859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:51.847226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 4859
I20260923 20:53:51.847429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:51.847987 388 external_mini_cluster.cc:1428] Setting key 77b4df17291214ee2f799192254606a8
I20260923 20:53:51.858623 4721 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
Sep 23 20:53:51 dist-test-slave-plcg krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:53:51.866869 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.848538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37381 (local address 127.0.97.62:35323)
0923 20:53:51.848775 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:51.848782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:51.850199 (+ 1417us) server_negotiation.cc:407] Connection header received
0923 20:53:51.852074 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:51.852079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:51.852153 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:51.852233 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:51.853802 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.854654 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.856393 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.856819 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.856940 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.860658 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:51.860664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:51.860667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:51.860698 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:51.863790 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:51.864614 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:51.864619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:51.864621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:51.864765 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:51.865183 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:51.865187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:51.865189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:51.865457 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:51.865584 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:51.866429 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 20:53:51.866604 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 20:53:51.867836 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.849055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35323 (local address 127.0.97.3:37381)
0923 20:53:51.850030 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:51.850077 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:51.851817 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:51.852415 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:51.852424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:51.852948 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:51.853601 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.853620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.854856 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.854863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.856231 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.856244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.857022 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.857026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.857212 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.857915 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:53:51.857943 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:51.860423 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:51.863956 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:51.863964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:51.863980 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:51.864468 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:51.864874 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:53:51.864877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:53:51.864879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:53:51.865084 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:53:51.865572 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:51.865579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:51.866002 (+ 423us) client_negotiation.cc:769] Sending connection context
0923 20:53:51.866268 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 20:53:51.866664 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":400,"threads_started":1}
I20260923 20:53:51.869427 4986 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35323
I20260923 20:53:51.869714 4986 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:51.870409 4986 heartbeater.cc:511] Master 127.0.97.62:35323 requested a full tablet report, sending...
I20260923 20:53:51.871851 4533 ts_manager.cc:194] Registered new tserver with Master: 1dcb67d744614f669a06279fa40243cd (127.0.97.3:39557)
I20260923 20:53:51.872686 4533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37381
I20260923 20:53:51.876066 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:53:51.897795 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.880611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.0.97.62:35323)
0923 20:53:51.880799 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:51.880804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:51.881333 (+ 529us) server_negotiation.cc:407] Connection header received
0923 20:53:51.881475 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:51.881479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:51.881541 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:51.881638 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:51.883579 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.884864 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.887439 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.888223 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.888383 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.891356 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:51.891363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:51.891369 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:51.891417 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:51.895305 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:51.896105 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:51.896109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:51.896112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:51.896237 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:51.896567 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:51.896570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:51.896572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:51.896867 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:51.897041 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:51.897378 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 20:53:51.897552 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:53:51.902475 4533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:51.909442 4533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:51.944453 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.922687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33779 (local address 127.0.0.1:60344)
0923 20:53:51.932691 (+ 10004us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:51.932755 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:51.932969 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:51.936555 (+ 3586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:51.936561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:51.936592 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:51.937003 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.937011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.938879 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.938886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.941376 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.941392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.943394 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.943403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.943604 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.943665 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 20:53:51.943795 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 20:53:51.944011 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9718,"thread_start_us":15118,"threads_started":1}
I20260923 20:53:51.944983 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.923976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.0.97.3:39557)
0923 20:53:51.935551 (+ 11575us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:51.935560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:51.935596 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:53:51.935704 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:51.935715 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:51.935956 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:51.936231 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:51.938354 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.939856 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.942402 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.943844 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.943969 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.944205 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:51.944458 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 20:53:51.944652 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11356,"thread_start_us":11106,"threads_started":1}
I20260923 20:53:51.945212 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.923113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39557 (local address 127.0.0.1:58494)
0923 20:53:51.935059 (+ 11946us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:51.935084 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:51.935210 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:51.936250 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:51.936258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:51.936290 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:51.936722 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.936738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.940018 (+ 3280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.940025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.942195 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.942209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.944059 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.944066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.944245 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.944280 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:53:51.944384 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:53:51.944533 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11808,"mutex_wait_us":6,"thread_start_us":11660,"threads_started":1}
I20260923 20:53:51.945662 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.926686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34835 (local address 127.0.0.1:58920)
0923 20:53:51.938139 (+ 11453us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:51.938158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:51.938292 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:51.938951 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:51.938955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:51.938995 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:51.939420 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.939431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.940949 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.940954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.943132 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:51.943152 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.944903 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.944910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:51.945080 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.945115 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:53:51.945218 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:53:51.945360 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11313,"thread_start_us":5520,"threads_started":1}
I20260923 20:53:51.946004 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.926904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.0.97.1:34835)
0923 20:53:51.937639 (+ 10735us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:51.937647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:51.938361 (+ 714us) server_negotiation.cc:407] Connection header received
0923 20:53:51.938488 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:51.938500 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:51.938768 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:51.938976 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:51.939588 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.940779 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.943312 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.944446 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.944554 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.944755 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:51.945467 (+ 712us) server_negotiation.cc:299] Negotiation successful
0923 20:53:51.945631 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10570,"thread_start_us":10373,"threads_started":1}
I20260923 20:53:51.946362 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:51.923250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.97.2:33779)
0923 20:53:51.935812 (+ 12562us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:51.935824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:51.935857 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:53:51.935969 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:51.935980 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:51.936260 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:51.936477 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:51.937168 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.938739 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.941623 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:51.943159 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:51.945407 (+ 2248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:51.945618 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:51.945699 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 20:53:51.946072 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12375,"thread_start_us":12253,"threads_started":1}
I20260923 20:53:51.949527 4656 tablet_service.cc:1467] Processing CreateTablet for tablet 9f401bb9962f4430a31fb7abf922dea2 (DEFAULT_TABLE table=test-table [id=84b6a8d613e14dee83c3f9e070f35c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:51.949712 4921 tablet_service.cc:1467] Processing CreateTablet for tablet 9f401bb9962f4430a31fb7abf922dea2 (DEFAULT_TABLE table=test-table [id=84b6a8d613e14dee83c3f9e070f35c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:51.949712 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 9f401bb9962f4430a31fb7abf922dea2 (DEFAULT_TABLE table=test-table [id=84b6a8d613e14dee83c3f9e070f35c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:51.950947 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f401bb9962f4430a31fb7abf922dea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:51.951113 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f401bb9962f4430a31fb7abf922dea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:51.951112 4921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f401bb9962f4430a31fb7abf922dea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:51.964099 5004 tablet_bootstrap.cc:492] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Bootstrap starting.
I20260923 20:53:51.966440 5006 tablet_bootstrap.cc:492] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Bootstrap starting.
I20260923 20:53:51.968278 5004 tablet_bootstrap.cc:654] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:51.969698 5004 log.cc:824] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:51.970184 5006 tablet_bootstrap.cc:654] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:51.971830 5006 log.cc:824] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:51.973399 5005 tablet_bootstrap.cc:492] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Bootstrap starting.
I20260923 20:53:51.976384 5006 tablet_bootstrap.cc:492] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: No bootstrap required, opened a new log
I20260923 20:53:51.976724 5006 ts_tablet_manager.cc:1397] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260923 20:53:51.977046 5004 tablet_bootstrap.cc:492] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: No bootstrap required, opened a new log
I20260923 20:53:51.977320 5005 tablet_bootstrap.cc:654] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:51.977340 5004 ts_tablet_manager.cc:1397] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260923 20:53:51.978698 5005 log.cc:824] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:51.984959 5006 raft_consensus.cc:348] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:51.985401 5006 raft_consensus.cc:374] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:51.985500 5006 raft_consensus.cc:729] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd08ba228d6f4f348ae4ffa8a8eb53f0, State: Initialized, Role: FOLLOWER
I20260923 20:53:51.986382 5006 consensus_queue.cc:260] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:51.987691 4854 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
I20260923 20:53:51.988291 5006 ts_tablet_manager.cc:1428] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260923 20:53:51.989787 5005 tablet_bootstrap.cc:492] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: No bootstrap required, opened a new log
I20260923 20:53:51.989671 5004 raft_consensus.cc:348] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:51.990126 5005 ts_tablet_manager.cc:1397] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20260923 20:53:51.990170 5004 raft_consensus.cc:374] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:51.990304 5004 raft_consensus.cc:729] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354e8b982d0740cfb9daf993abcbaac6, State: Initialized, Role: FOLLOWER
I20260923 20:53:51.990936 5004 consensus_queue.cc:260] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:51.992835 5004 ts_tablet_manager.cc:1428] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260923 20:53:51.999523 5005 raft_consensus.cc:348] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:51.999966 5005 raft_consensus.cc:374] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:52.000102 5005 raft_consensus.cc:729] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcb67d744614f669a06279fa40243cd, State: Initialized, Role: FOLLOWER
I20260923 20:53:52.000721 5005 consensus_queue.cc:260] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:52.001806 4986 heartbeater.cc:503] Master 127.0.97.62:35323 was elected leader, sending a full tablet report...
I20260923 20:53:52.002403 5005 ts_tablet_manager.cc:1428] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260923 20:53:52.075237 4722 tablet.cc:2369] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:53:52.088343 4855 tablet.cc:2369] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:53:52.097127 4987 tablet.cc:2369] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:53:52.379730 5010 raft_consensus.cc:482] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:52.380093 5010 raft_consensus.cc:504] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:52.381901 5010 leader_election.cc:288] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dcb67d744614f669a06279fa40243cd (127.0.97.3:39557), 354e8b982d0740cfb9daf993abcbaac6 (127.0.97.1:34835)
I20260923 20:53:52.389375 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.382358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52913 (local address 127.0.97.1:34835)
0923 20:53:52.382540 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:52.382547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:52.383345 (+ 798us) server_negotiation.cc:407] Connection header received
0923 20:53:52.383514 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:52.383520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:52.383599 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:52.383705 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:52.384515 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.385515 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.387582 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.388364 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.388516 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.388639 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:52.388967 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 20:53:52.389103 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:53:52.389385 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.382358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46053 (local address 127.0.97.3:39557)
0923 20:53:52.382540 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:52.382547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:52.383345 (+ 798us) server_negotiation.cc:407] Connection header received
0923 20:53:52.383514 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:52.383520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:52.383599 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:52.383705 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:53:52.384523 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.385416 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.387627 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.388392 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.388555 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.388670 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:52.389019 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 20:53:52.389142 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:53:52.389564 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.382233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39557 (local address 127.0.97.2:46053)
0923 20:53:52.383131 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:52.383158 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:52.383342 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:52.383754 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:52.383760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:52.383791 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:52.384361 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:52.384369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.385567 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.385575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:52.387402 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:52.387417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.388672 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.388679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:52.388835 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.388858 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:53:52.388973 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:53:52.389103 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":119,"thread_start_us":493,"threads_started":1}
I20260923 20:53:52.389643 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.382286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34835 (local address 127.0.97.2:52913)
0923 20:53:52.383190 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:52.383206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:52.383376 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:52.383754 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:52.383760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:52.383802 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:53:52.384298 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:52.384316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.385662 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.385666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:52.387385 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:52.387401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.388622 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.388629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:52.388797 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.388822 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:53:52.388901 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 20:53:52.389059 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":53,"spinlock_wait_cycles":56448,"thread_start_us":500,"threads_started":1}
I20260923 20:53:52.390568 4676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f401bb9962f4430a31fb7abf922dea2" candidate_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "354e8b982d0740cfb9daf993abcbaac6" is_pre_election: true
I20260923 20:53:52.390652 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f401bb9962f4430a31fb7abf922dea2" candidate_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcb67d744614f669a06279fa40243cd" is_pre_election: true
I20260923 20:53:52.391094 4941 raft_consensus.cc:2454] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd08ba228d6f4f348ae4ffa8a8eb53f0 in term 0.
I20260923 20:53:52.391093 4676 raft_consensus.cc:2454] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd08ba228d6f4f348ae4ffa8a8eb53f0 in term 0.
I20260923 20:53:52.391762 4744 leader_election.cc:302] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354e8b982d0740cfb9daf993abcbaac6, dd08ba228d6f4f348ae4ffa8a8eb53f0; no voters:
I20260923 20:53:52.392203 5010 raft_consensus.cc:2781] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:52.392359 5010 raft_consensus.cc:482] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:52.392493 5010 raft_consensus.cc:3037] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:52.394909 5010 raft_consensus.cc:504] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:52.395538 5010 leader_election.cc:288] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [CANDIDATE]: Term 1 election: Requested vote from peers 1dcb67d744614f669a06279fa40243cd (127.0.97.3:39557), 354e8b982d0740cfb9daf993abcbaac6 (127.0.97.1:34835)
I20260923 20:53:52.396076 4941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f401bb9962f4430a31fb7abf922dea2" candidate_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcb67d744614f669a06279fa40243cd"
I20260923 20:53:52.396154 4676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f401bb9962f4430a31fb7abf922dea2" candidate_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "354e8b982d0740cfb9daf993abcbaac6"
I20260923 20:53:52.396306 4941 raft_consensus.cc:3037] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:52.396440 4676 raft_consensus.cc:3037] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:52.398814 4941 raft_consensus.cc:2454] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd08ba228d6f4f348ae4ffa8a8eb53f0 in term 1.
I20260923 20:53:52.399026 4676 raft_consensus.cc:2454] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd08ba228d6f4f348ae4ffa8a8eb53f0 in term 1.
I20260923 20:53:52.399371 4745 leader_election.cc:302] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcb67d744614f669a06279fa40243cd, dd08ba228d6f4f348ae4ffa8a8eb53f0; no voters:
I20260923 20:53:52.399737 5010 raft_consensus.cc:2781] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:52.400218 5010 raft_consensus.cc:686] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 1 LEADER]: Becoming Leader. State: Replica: dd08ba228d6f4f348ae4ffa8a8eb53f0, State: Running, Role: LEADER
I20260923 20:53:52.400763 5010 consensus_queue.cc:237] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } }
I20260923 20:53:52.406033 4533 catalog_manager.cc:5495] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd08ba228d6f4f348ae4ffa8a8eb53f0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08ba228d6f4f348ae4ffa8a8eb53f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:52.508314 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.502220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.0.97.2:33779)
0923 20:53:52.503063 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:52.503070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:52.503096 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:53:52.503260 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:52.503265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:52.503342 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:52.503486 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:53:52.503920 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.504637 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.506145 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.506549 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.506648 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.507146 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:53:52.507671 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:53:52.507788 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:52.507910 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 20:53:52.507978 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":881,"threads_started":1}
I20260923 20:53:52.516419 4941 raft_consensus.cc:1261] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 1 FOLLOWER]: Refusing update from remote peer dd08ba228d6f4f348ae4ffa8a8eb53f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:52.516419 4676 raft_consensus.cc:1261] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 1 FOLLOWER]: Refusing update from remote peer dd08ba228d6f4f348ae4ffa8a8eb53f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:52.517498 5010 consensus_queue.cc:1035] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "354e8b982d0740cfb9daf993abcbaac6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:52.517803 5015 consensus_queue.cc:1035] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dcb67d744614f669a06279fa40243cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:52.531776 5024 mvcc.cc:205] Tried to move back new op lower bound from 7332646225977155584 to 7332646225517551616. Current Snapshot: MvccSnapshot[applied={T|T < 7332646225977155584}]
I20260923 20:53:52.544492 4532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:52.544806 4532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:52.548625 4532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be006687ad2244b18accc9a0f68dfe94: Sending DeleteTablet for 3 replicas of tablet 9f401bb9962f4430a31fb7abf922dea2
I20260923 20:53:52.549899 4921 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f401bb9962f4430a31fb7abf922dea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58494
I20260923 20:53:52.549899 4656 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f401bb9962f4430a31fb7abf922dea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58920
I20260923 20:53:52.549899 4789 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f401bb9962f4430a31fb7abf922dea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60344
I20260923 20:53:52.552647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 4594
I20260923 20:53:52.562312 5032 tablet_replica.cc:330] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: stopping tablet replica
I20260923 20:53:52.562997 5033 tablet_replica.cc:330] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: stopping tablet replica
I20260923 20:53:52.563656 5034 tablet_replica.cc:330] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: stopping tablet replica
I20260923 20:53:52.563628 5032 raft_consensus.cc:2229] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:52.564217 5032 raft_consensus.cc:2258] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:52.564553 5033 raft_consensus.cc:2229] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:52.564771 5034 raft_consensus.cc:2229] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:52.565577 5034 raft_consensus.cc:2258] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:52.566135 5033 raft_consensus.cc:2258] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:52.566494 5032 ts_tablet_manager.cc:1905] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:52.569195 5034 ts_tablet_manager.cc:1905] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:52.571098 5033 ts_tablet_manager.cc:1905] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:53:52 dist-test-slave-plcg krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:53:52.574306 5032 ts_tablet_manager.cc:1918] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:52.574486 5032 log.cc:1196] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/9f401bb9962f4430a31fb7abf922dea2
I20260923 20:53:52.575026 5032 ts_tablet_manager.cc:1939] T 9f401bb9962f4430a31fb7abf922dea2 P 1dcb67d744614f669a06279fa40243cd: Deleting consensus metadata
I20260923 20:53:52.577373 4520 catalog_manager.cc:4841] TS 1dcb67d744614f669a06279fa40243cd (127.0.97.3:39557): tablet 9f401bb9962f4430a31fb7abf922dea2 (table test-table [id=84b6a8d613e14dee83c3f9e070f35c1d]) successfully deleted
I20260923 20:53:52.582286 5033 ts_tablet_manager.cc:1918] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:52.582573 5033 log.cc:1196] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/9f401bb9962f4430a31fb7abf922dea2
I20260923 20:53:52.583215 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.554618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.0.97.1:34835)
0923 20:53:52.554875 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:52.554885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:52.563382 (+ 8497us) server_negotiation.cc:407] Connection header received
0923 20:53:52.563682 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:52.563689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:52.563818 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:52.563904 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:52.565969 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.567189 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.569109 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.569763 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.569914 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.575747 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:52.575765 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:52.575795 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:52.575844 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:52.579458 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:52.580260 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:52.580277 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:52.580283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:52.580503 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:52.581003 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:52.581008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:52.581011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:52.581517 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:52.581878 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:52.582508 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 20:53:52.582774 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:53:52.583774 5033 ts_tablet_manager.cc:1939] T 9f401bb9962f4430a31fb7abf922dea2 P 354e8b982d0740cfb9daf993abcbaac6: Deleting consensus metadata
I20260923 20:53:52.584474 4716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:52.584678 5034 ts_tablet_manager.cc:1918] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:52.584945 5034 log.cc:1196] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/9f401bb9962f4430a31fb7abf922dea2
I20260923 20:53:52.585711 5034 ts_tablet_manager.cc:1939] T 9f401bb9962f4430a31fb7abf922dea2 P dd08ba228d6f4f348ae4ffa8a8eb53f0: Deleting consensus metadata
I20260923 20:53:52.588032 4518 catalog_manager.cc:4841] TS dd08ba228d6f4f348ae4ffa8a8eb53f0 (127.0.97.2:33779): tablet 9f401bb9962f4430a31fb7abf922dea2 (table test-table [id=84b6a8d613e14dee83c3f9e070f35c1d]) successfully deleted
I20260923 20:53:52.701263 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 4594
I20260923 20:53:52.701279 4519 catalog_manager.cc:4841] TS 354e8b982d0740cfb9daf993abcbaac6 (127.0.97.1:34835): tablet 9f401bb9962f4430a31fb7abf922dea2 (table test-table [id=84b6a8d613e14dee83c3f9e070f35c1d]) successfully deleted
I20260923 20:53:52.723882 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 4727
Sep 23 20:53:52 dist-test-slave-plcg krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:53:52.741766 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.724936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.0.97.2:33779)
0923 20:53:52.725139 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:52.725146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:52.725167 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:53:52.725231 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:52.725237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:52.725314 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:52.725434 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:52.727057 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.728260 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.729479 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.730179 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.730312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.734738 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:52.734759 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:52.734783 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:52.734835 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:52.738829 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:52.739575 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:52.739584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:52.739589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:52.739746 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:52.740099 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:52.740104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:52.740108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:52.740568 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:52.740915 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:52.741106 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 20:53:52.741366 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:53:52.742764 4849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:52.856302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 4727
I20260923 20:53:52.881919 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 4859
Sep 23 20:53:52 dist-test-slave-plcg krb5kdc[4489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:53:52.899346 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:52.882967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.0.97.3:39557)
0923 20:53:52.883171 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:52.883190 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:52.883215 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:53:52.883290 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:52.883294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:52.883372 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:52.883518 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:52.884892 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.885695 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.887308 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:52.887755 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:52.887914 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:52.892782 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:52.892800 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:52.892818 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:53:52.892861 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:52.896068 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:52.896934 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:52.896942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:52.896944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:52.897094 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:53:52.897539 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:53:52.897546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:53:52.897550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:53:52.898042 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:52.898373 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:52.898721 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 20:53:52.898950 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:53:52.900066 4981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:52.990928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 4859
I20260923 20:53:53.007998 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 4502
I20260923 20:53:53.009191 4563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:53.115662 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 4502
2026-09-23T20:53:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4699 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T20:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:53Z Disabled control of system clock
I20260923 20:53:53.186129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35969
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:53:53.524938 5043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:53.525254 5043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:53.525341 5043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:53.532678 5043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:53.532807 5043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:53.532846 5043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:53.532895 5043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:53.532940 5043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:53:53.542474 5043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:53.543745 5043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:53.545599 5043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:53.557884 5048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:53.557917 5049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:53.558501 5051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:53.558959 5043 server_base.cc:1034] running on GCE node
I20260923 20:53:53.559787 5043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:53.561316 5043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:53.562564 5043 hybrid_clock.cc:648] HybridClock initialized: now 1790196833562526 us; error 53 us; skew 500 ppm
I20260923 20:53:53.562950 5043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:53.571225 5043 webserver.cc:460] Webserver started at http://127.0.97.62:33303/ using document root <none> and password file <none>
I20260923 20:53:53.572019 5043 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:53.572165 5043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:53.572517 5043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:53.575587 5043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "11515010d05d4d9fafd75c423a77e39d"
format_stamp: "Formatted at 2026-09-23 20:53:53 on dist-test-slave-plcg"
server_key: "e042204bf83765d4856225ecbe9d62c9"
server_key_iv: "f7f50b2592411f4ebd8121b365a316b1"
server_key_version: "encryptionkey@0"
I20260923 20:53:53.576436 5043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "11515010d05d4d9fafd75c423a77e39d"
format_stamp: "Formatted at 2026-09-23 20:53:53 on dist-test-slave-plcg"
server_key: "e042204bf83765d4856225ecbe9d62c9"
server_key_iv: "f7f50b2592411f4ebd8121b365a316b1"
server_key_version: "encryptionkey@0"
I20260923 20:53:53.581928 5043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:53:53.585551 5057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:53.587112 5043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:53:53.587311 5043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "11515010d05d4d9fafd75c423a77e39d"
format_stamp: "Formatted at 2026-09-23 20:53:53 on dist-test-slave-plcg"
server_key: "e042204bf83765d4856225ecbe9d62c9"
server_key_iv: "f7f50b2592411f4ebd8121b365a316b1"
server_key_version: "encryptionkey@0"
I20260923 20:53:53.587561 5043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:53.593622 5043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:53.594525 5043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:53.594839 5043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:53.613289 5043 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35969
I20260923 20:53:53.613312 5108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35969 every 8 connection(s)
I20260923 20:53:53.614704 5043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:53.616729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5043
I20260923 20:53:53.616933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:53.617743 388 external_mini_cluster.cc:1428] Setting key ca680a61d21d4ffeaf480fc694b748e3
I20260923 20:53:53.619365 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:53.626361 5112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:53.619855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.0.97.62:35969)
0923 20:53:53.620819 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:53.620831 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:53.620871 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:53:53.622487 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:53.622509 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:53.622940 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:53.623378 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:53.624540 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:53.624546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:53.624559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:53.624575 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:53.624652 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:53.624667 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:53.624913 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:53.625083 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 20:53:53.625337 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":350,"threads_started":1}
I20260923 20:53:53.631814 5109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d: Bootstrap starting.
I20260923 20:53:53.635223 5109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:53.636345 5109 log.cc:824] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:53.639288 5109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d: No bootstrap required, opened a new log
I20260923 20:53:53.646930 5109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11515010d05d4d9fafd75c423a77e39d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260923 20:53:53.647342 5109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:53.647449 5109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11515010d05d4d9fafd75c423a77e39d, State: Initialized, Role: FOLLOWER
I20260923 20:53:53.648115 5109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11515010d05d4d9fafd75c423a77e39d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260923 20:53:53.648352 5109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:53.648489 5109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:53.648669 5109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:53.650945 5109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11515010d05d4d9fafd75c423a77e39d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260923 20:53:53.651512 5109 leader_election.cc:302] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11515010d05d4d9fafd75c423a77e39d; no voters:
I20260923 20:53:53.652176 5109 leader_election.cc:288] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:53.652375 5114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:53.653196 5114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [term 1 LEADER]: Becoming Leader. State: Replica: 11515010d05d4d9fafd75c423a77e39d, State: Running, Role: LEADER
I20260923 20:53:53.653764 5114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11515010d05d4d9fafd75c423a77e39d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260923 20:53:53.654657 5109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:53.656911 5115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11515010d05d4d9fafd75c423a77e39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11515010d05d4d9fafd75c423a77e39d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } } }
I20260923 20:53:53.657104 5116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11515010d05d4d9fafd75c423a77e39d. Latest consensus state: current_term: 1 leader_uuid: "11515010d05d4d9fafd75c423a77e39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11515010d05d4d9fafd75c423a77e39d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } } }
I20260923 20:53:53.657435 5116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:53.657444 5115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:53.659375 5124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:53.665187 5124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:53.675016 5124 catalog_manager.cc:1347] Generated new cluster ID: 7680cf95e4ad4a74a263de8124134f60
I20260923 20:53:53.675154 5124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:53.690623 5124 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:53.691242 5124 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:53.715444 5124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d: Generated new TSK 0
I20260923 20:53:53.716629 5124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:53:53.725474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--builtin_ntp_servers=127.0.97.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:53:54.070941 5133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:54.071267 5133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:54.071359 5133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:54.078899 5133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:54.079018 5133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:54.079130 5133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:53:54.088804 5133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:54.090191 5133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:54.092180 5133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:54.106415 5141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:54.106492 5139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:54.106494 5138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:54.107465 5133 server_base.cc:1034] running on GCE node
I20260923 20:53:54.108103 5133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:54.109094 5133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:54.110302 5133 hybrid_clock.cc:648] HybridClock initialized: now 1790196834110262 us; error 44 us; skew 500 ppm
I20260923 20:53:54.110654 5133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:54.113240 5133 webserver.cc:460] Webserver started at http://127.0.97.1:43505/ using document root <none> and password file <none>
I20260923 20:53:54.114101 5133 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:54.114234 5133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:54.114557 5133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:54.117585 5133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "9f2cd6979e7c418cb84302aab7ff74d9"
format_stamp: "Formatted at 2026-09-23 20:53:54 on dist-test-slave-plcg"
server_key: "894a609f5713f3b8b5e6a3252480d7ba"
server_key_iv: "0ed95d7985d66cc8bca99d1c0ae92a36"
server_key_version: "encryptionkey@0"
I20260923 20:53:54.118567 5133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9f2cd6979e7c418cb84302aab7ff74d9"
format_stamp: "Formatted at 2026-09-23 20:53:54 on dist-test-slave-plcg"
server_key: "894a609f5713f3b8b5e6a3252480d7ba"
server_key_iv: "0ed95d7985d66cc8bca99d1c0ae92a36"
server_key_version: "encryptionkey@0"
I20260923 20:53:54.124289 5133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:53:54.128088 5147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:54.129545 5133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:53:54.129750 5133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "9f2cd6979e7c418cb84302aab7ff74d9"
format_stamp: "Formatted at 2026-09-23 20:53:54 on dist-test-slave-plcg"
server_key: "894a609f5713f3b8b5e6a3252480d7ba"
server_key_iv: "0ed95d7985d66cc8bca99d1c0ae92a36"
server_key_version: "encryptionkey@0"
I20260923 20:53:54.130055 5133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:54.136196 5133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:54.137353 5133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:54.137694 5133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:54.138800 5133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:54.140329 5133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:54.140456 5133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:54.140566 5133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:54.140658 5133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:54.179271 5133 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39639
I20260923 20:53:54.179323 5259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39639 every 8 connection(s)
I20260923 20:53:54.180716 5133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:53:54.185973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5133
I20260923 20:53:54.186182 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:53:54.186792 388 external_mini_cluster.cc:1428] Setting key a3604ab57d39d9929fcc890f0eaafd90
I20260923 20:53:54.189666 5262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:54.183431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52437 (local address 127.0.97.62:35969)
0923 20:53:54.183888 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:54.183892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:54.185095 (+ 1203us) server_negotiation.cc:407] Connection header received
0923 20:53:54.187123 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:54.187128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:54.187203 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:54.187332 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:54.188858 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:54.188864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:54.188868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:54.188894 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:54.188937 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:54.188949 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:54.189082 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:54.189281 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 20:53:54.189347 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260923 20:53:54.189884 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--builtin_ntp_servers=127.0.97.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:53:54.190858 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:54.183947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.1:52437)
0923 20:53:54.184912 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:54.184952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:54.186763 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:54.187611 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:54.187620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:54.187755 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:54.188540 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:54.188543 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:54.188590 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:54.188688 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:54.189055 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:54.189069 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:54.189077 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:53:54.189382 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 20:53:54.189689 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":302,"threads_started":1}
I20260923 20:53:54.192829 5260 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35969
I20260923 20:53:54.193182 5260 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:54.193862 5260 heartbeater.cc:511] Master 127.0.97.62:35969 requested a full tablet report, sending...
I20260923 20:53:54.196153 5074 ts_manager.cc:194] Registered new tserver with Master: 9f2cd6979e7c418cb84302aab7ff74d9 (127.0.97.1:39639)
W20260923 20:53:54.533023 5264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:54.533331 5264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:54.533424 5264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:54.541080 5264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:54.541204 5264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:54.541316 5264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:53:54.550720 5264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:54.552038 5264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:54.553889 5264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:54.568674 5269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:54.568769 5270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:54.568718 5272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:54.569748 5264 server_base.cc:1034] running on GCE node
I20260923 20:53:54.570518 5264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:54.571678 5264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:54.572897 5264 hybrid_clock.cc:648] HybridClock initialized: now 1790196834572831 us; error 73 us; skew 500 ppm
I20260923 20:53:54.573248 5264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:54.575765 5264 webserver.cc:460] Webserver started at http://127.0.97.2:39467/ using document root <none> and password file <none>
I20260923 20:53:54.576532 5264 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:54.576668 5264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:54.577013 5264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:54.580085 5264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c"
format_stamp: "Formatted at 2026-09-23 20:53:54 on dist-test-slave-plcg"
server_key: "ccf17239d565fb5ebb6ec2e048abee8c"
server_key_iv: "6ea87097fdad05e6b551cf71c1c9f8d2"
server_key_version: "encryptionkey@0"
I20260923 20:53:54.580961 5264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c"
format_stamp: "Formatted at 2026-09-23 20:53:54 on dist-test-slave-plcg"
server_key: "ccf17239d565fb5ebb6ec2e048abee8c"
server_key_iv: "6ea87097fdad05e6b551cf71c1c9f8d2"
server_key_version: "encryptionkey@0"
I20260923 20:53:54.586634 5264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:53:54.590574 5278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:54.592363 5264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:53:54.592561 5264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c"
format_stamp: "Formatted at 2026-09-23 20:53:54 on dist-test-slave-plcg"
server_key: "ccf17239d565fb5ebb6ec2e048abee8c"
server_key_iv: "6ea87097fdad05e6b551cf71c1c9f8d2"
server_key_version: "encryptionkey@0"
I20260923 20:53:54.592828 5264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:54.599086 5264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:54.600015 5264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:54.600311 5264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:54.601320 5264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:54.602905 5264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:54.603037 5264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:54.603186 5264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:54.603276 5264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:54.638926 5264 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41975
I20260923 20:53:54.638959 5390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41975 every 8 connection(s)
I20260923 20:53:54.640619 5264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:53:54.641971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5264
I20260923 20:53:54.642139 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:53:54.642921 388 external_mini_cluster.cc:1428] Setting key e6db5813ff4fd1749144e8ca6281c4a6
I20260923 20:53:54.646443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--builtin_ntp_servers=127.0.97.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:53:54.650456 5262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:54.645207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50241 (local address 127.0.97.62:35969)
0923 20:53:54.645390 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:54.645397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:54.645973 (+ 576us) server_negotiation.cc:407] Connection header received
0923 20:53:54.647856 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:54.647865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:54.647939 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:54.648062 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:54.649513 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:54.649522 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:54.649529 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:54.649556 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:54.649604 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:54.649624 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:54.649770 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:54.650019 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 20:53:54.650095 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:53:54.651415 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:54.644602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.2:50241)
0923 20:53:54.645771 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:54.645847 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:54.647515 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:54.648287 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:54.648295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:54.648402 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:54.649168 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:54.649170 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:54.649212 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:54.649309 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:54.649743 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:54.649750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:54.649756 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:53:54.650058 (+ 302us) client_negotiation.cc:240] Negotiation successful
0923 20:53:54.650291 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":431,"threads_started":1}
I20260923 20:53:54.653057 5391 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35969
I20260923 20:53:54.653378 5391 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:54.654072 5391 heartbeater.cc:511] Master 127.0.97.62:35969 requested a full tablet report, sending...
I20260923 20:53:54.655697 5074 ts_manager.cc:194] Registered new tserver with Master: 0373c36eb5ff4d82b178b47f1f2bcf1c (127.0.97.2:41975)
W20260923 20:53:54.995046 5394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:54.995409 5394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:54.995575 5394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:55.003357 5394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:55.003531 5394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:55.003712 5394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:53:55.013408 5394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:55.014830 5394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:55.016700 5394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:55.030478 5399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:55.030388 5402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:55.030546 5400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:55.031530 5394 server_base.cc:1034] running on GCE node
I20260923 20:53:55.032194 5394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:55.033370 5394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:55.034651 5394 hybrid_clock.cc:648] HybridClock initialized: now 1790196835034602 us; error 49 us; skew 500 ppm
I20260923 20:53:55.035013 5394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:55.037482 5394 webserver.cc:460] Webserver started at http://127.0.97.3:44083/ using document root <none> and password file <none>
I20260923 20:53:55.038304 5394 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:55.038444 5394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:55.038774 5394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:55.041740 5394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "b5b4f643bb594521b971fd38aeafdcf1"
format_stamp: "Formatted at 2026-09-23 20:53:55 on dist-test-slave-plcg"
server_key: "64fab0958796fef34a2f157162e37699"
server_key_iv: "a2f6dfe5df5164b2133de98305f718c7"
server_key_version: "encryptionkey@0"
I20260923 20:53:55.042703 5394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b5b4f643bb594521b971fd38aeafdcf1"
format_stamp: "Formatted at 2026-09-23 20:53:55 on dist-test-slave-plcg"
server_key: "64fab0958796fef34a2f157162e37699"
server_key_iv: "a2f6dfe5df5164b2133de98305f718c7"
server_key_version: "encryptionkey@0"
I20260923 20:53:55.048341 5394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:55.052253 5408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:55.053915 5394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:53:55.054128 5394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "b5b4f643bb594521b971fd38aeafdcf1"
format_stamp: "Formatted at 2026-09-23 20:53:55 on dist-test-slave-plcg"
server_key: "64fab0958796fef34a2f157162e37699"
server_key_iv: "a2f6dfe5df5164b2133de98305f718c7"
server_key_version: "encryptionkey@0"
I20260923 20:53:55.054440 5394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:55.060883 5394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:55.061779 5394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:55.062125 5394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:55.063097 5394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:53:55.064589 5394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:53:55.064718 5394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:55.064860 5394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:53:55.064949 5394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:55.099903 5394 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33827
I20260923 20:53:55.099925 5520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33827 every 8 connection(s)
I20260923 20:53:55.101333 5394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:53:55.108964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5394
I20260923 20:53:55.109128 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:53:55.109547 388 external_mini_cluster.cc:1428] Setting key 4ed09abfadbcd4d960053f5b48c95cb3
I20260923 20:53:55.109546 5262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.104069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58961 (local address 127.0.97.62:35969)
0923 20:53:55.104243 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.104248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.105678 (+ 1430us) server_negotiation.cc:407] Connection header received
0923 20:53:55.107387 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.107391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.107446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.107536 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.108777 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.108783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.108788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.108812 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.108840 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.108851 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.108973 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.109211 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.109279 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:53:55.110690 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.104603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.3:58961)
0923 20:53:55.105533 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.105579 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:55.107119 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:55.107740 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:55.107749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:55.107845 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:55.108467 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:55.108469 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:55.108506 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:55.108589 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:55.108954 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:55.108963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:55.108971 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:53:55.109301 (+ 330us) client_negotiation.cc:240] Negotiation successful
0923 20:53:55.109566 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":291,"threads_started":1}
I20260923 20:53:55.112056 5521 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35969
I20260923 20:53:55.112387 5521 heartbeater.cc:464] Registering TS with master...
I20260923 20:53:55.112999 5521 heartbeater.cc:511] Master 127.0.97.62:35969 requested a full tablet report, sending...
I20260923 20:53:55.114511 5074 ts_manager.cc:194] Registered new tserver with Master: b5b4f643bb594521b971fd38aeafdcf1 (127.0.97.3:33827)
I20260923 20:53:55.124931 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:53:55.132269 5262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.129683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.0.97.62:35969)
0923 20:53:55.130055 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.130065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.130469 (+ 404us) server_negotiation.cc:407] Connection header received
0923 20:53:55.130608 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.130612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.130666 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.130769 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.131632 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.131636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.131640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.131654 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.131685 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.131696 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.131819 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.131914 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.131961 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260923 20:53:55.136409 5074 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:53:55.143422 5074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:53:55.173784 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.160460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.0.97.2:41975)
0923 20:53:55.168945 (+ 8485us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.168957 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.169427 (+ 470us) server_negotiation.cc:407] Connection header received
0923 20:53:55.169601 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.169612 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.169947 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.170875 (+ 928us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.172355 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.172368 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.172386 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.172415 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.172478 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.172502 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.173270 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.173347 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.173453 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8268,"thread_start_us":8058,"threads_started":1}
I20260923 20:53:55.174345 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.157125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41975 (local address 127.0.0.1:57350)
0923 20:53:55.169286 (+ 12161us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.169302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:55.169451 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:55.170195 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:55.170202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:55.170314 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:55.172042 (+ 1728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:55.172046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:55.172079 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:55.172164 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:55.172622 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:55.172635 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:55.172647 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:53:55.172835 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 20:53:55.172927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12010,"threads_started":1}
I20260923 20:53:55.175037 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.156586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.97.3:33827)
0923 20:53:55.170846 (+ 14260us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.170854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.170890 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:53:55.170964 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.170972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.171185 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.171399 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.173653 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.173661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.173674 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.173691 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.173742 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.173768 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.173980 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.174516 (+ 536us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.174617 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14079,"thread_start_us":13920,"threads_started":1}
I20260923 20:53:55.175043 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.156391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33827 (local address 127.0.0.1:60796)
0923 20:53:55.165748 (+ 9357us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.165865 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:55.166075 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:55.171438 (+ 5363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:55.171446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:55.171483 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:55.172539 (+ 1056us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:55.172545 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:55.172571 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:55.172635 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:55.174214 (+ 1579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:55.174222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:55.174238 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 20:53:55.174601 (+ 363us) client_negotiation.cc:240] Negotiation successful
0923 20:53:55.174653 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8961,"thread_start_us":12602,"threads_started":1}
I20260923 20:53:55.178261 5326 tablet_service.cc:1467] Processing CreateTablet for tablet dc503ab2b2904221812d6dbe41b1266b (DEFAULT_TABLE table=test-table [id=aaaaff970e2a45b48f2d9c52c6a3479f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:55.179275 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.158641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40036)
0923 20:53:55.174499 (+ 15858us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.174520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:55.174727 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:55.177056 (+ 2329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:55.177062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:55.177090 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:55.178016 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:55.178019 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:55.178036 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:55.178082 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:55.178599 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:55.178604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:55.178609 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:53:55.178774 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 20:53:55.178831 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15718,"thread_start_us":6478,"threads_started":1}
I20260923 20:53:55.179090 5456 tablet_service.cc:1467] Processing CreateTablet for tablet dc503ab2b2904221812d6dbe41b1266b (DEFAULT_TABLE table=test-table [id=aaaaff970e2a45b48f2d9c52c6a3479f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:55.179787 5326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc503ab2b2904221812d6dbe41b1266b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:55.179992 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.157345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.0.97.1:39639)
0923 20:53:55.170321 (+ 12976us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.170329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.174627 (+ 4298us) server_negotiation.cc:407] Connection header received
0923 20:53:55.174908 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.174917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.175154 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.175284 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.178270 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.178278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.178291 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.178319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.178376 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.178420 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.178546 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.179492 (+ 946us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.179593 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12808,"thread_start_us":12678,"threads_started":1}
I20260923 20:53:55.180755 5456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc503ab2b2904221812d6dbe41b1266b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:55.184338 5195 tablet_service.cc:1467] Processing CreateTablet for tablet dc503ab2b2904221812d6dbe41b1266b (DEFAULT_TABLE table=test-table [id=aaaaff970e2a45b48f2d9c52c6a3479f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:53:55.185802 5195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc503ab2b2904221812d6dbe41b1266b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:55.195304 5539 tablet_bootstrap.cc:492] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Bootstrap starting.
I20260923 20:53:55.197634 5540 tablet_bootstrap.cc:492] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Bootstrap starting.
I20260923 20:53:55.199644 5539 tablet_bootstrap.cc:654] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:55.199890 5260 heartbeater.cc:503] Master 127.0.97.62:35969 was elected leader, sending a full tablet report...
I20260923 20:53:55.201637 5539 log.cc:824] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:55.202298 5540 tablet_bootstrap.cc:654] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:55.203918 5540 log.cc:824] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:55.204895 5541 tablet_bootstrap.cc:492] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: Bootstrap starting.
I20260923 20:53:55.208392 5540 tablet_bootstrap.cc:492] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: No bootstrap required, opened a new log
I20260923 20:53:55.208707 5540 ts_tablet_manager.cc:1397] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 20:53:55.208722 5539 tablet_bootstrap.cc:492] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: No bootstrap required, opened a new log
I20260923 20:53:55.208907 5541 tablet_bootstrap.cc:654] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:55.209007 5539 ts_tablet_manager.cc:1397] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260923 20:53:55.210299 5541 log.cc:824] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:55.217017 5541 tablet_bootstrap.cc:492] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: No bootstrap required, opened a new log
I20260923 20:53:55.216929 5540 raft_consensus.cc:348] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.217342 5541 ts_tablet_manager.cc:1397] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260923 20:53:55.217407 5540 raft_consensus.cc:374] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:55.217519 5540 raft_consensus.cc:729] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b4f643bb594521b971fd38aeafdcf1, State: Initialized, Role: FOLLOWER
I20260923 20:53:55.218227 5540 consensus_queue.cc:260] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.219584 5521 heartbeater.cc:503] Master 127.0.97.62:35969 was elected leader, sending a full tablet report...
I20260923 20:53:55.220337 5540 ts_tablet_manager.cc:1428] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260923 20:53:55.220762 5539 raft_consensus.cc:348] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.221243 5539 raft_consensus.cc:374] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:55.221339 5539 raft_consensus.cc:729] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0373c36eb5ff4d82b178b47f1f2bcf1c, State: Initialized, Role: FOLLOWER
I20260923 20:53:55.222101 5539 consensus_queue.cc:260] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.224064 5391 heartbeater.cc:503] Master 127.0.97.62:35969 was elected leader, sending a full tablet report...
I20260923 20:53:55.224799 5539 ts_tablet_manager.cc:1428] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260923 20:53:55.228076 5541 raft_consensus.cc:348] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.228607 5541 raft_consensus.cc:374] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:55.228747 5541 raft_consensus.cc:729] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2cd6979e7c418cb84302aab7ff74d9, State: Initialized, Role: FOLLOWER
I20260923 20:53:55.229453 5541 consensus_queue.cc:260] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.231208 5541 ts_tablet_manager.cc:1428] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260923 20:53:55.240895 5547 raft_consensus.cc:482] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:53:55.241236 5547 raft_consensus.cc:504] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.243050 5547 leader_election.cc:288] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5b4f643bb594521b971fd38aeafdcf1 (127.0.97.3:33827), 0373c36eb5ff4d82b178b47f1f2bcf1c (127.0.97.2:41975)
I20260923 20:53:55.246821 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.243375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56209 (local address 127.0.97.3:33827)
0923 20:53:55.243595 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.243605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.244719 (+ 1114us) server_negotiation.cc:407] Connection header received
0923 20:53:55.244905 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.244909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.244956 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.245060 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.246173 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.246179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.246190 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.246205 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.246241 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.246251 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.246380 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.246472 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.246537 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:53:55.246914 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.243724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53385 (local address 127.0.97.2:41975)
0923 20:53:55.244038 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.244048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.244620 (+ 572us) server_negotiation.cc:407] Connection header received
0923 20:53:55.244837 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.244841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.244889 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.244995 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.246271 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.246277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.246281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.246300 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.246337 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.246349 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.246467 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.246559 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.246624 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260923 20:53:55.247179 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.243446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33827 (local address 127.0.97.1:56209)
0923 20:53:55.244604 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.244619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:55.244767 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:55.245078 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:55.245083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:55.245106 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:55.245972 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:55.245975 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:55.245989 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:55.246031 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:55.246351 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:55.246358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:55.246363 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:53:55.246508 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 20:53:55.246557 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":595,"threads_started":1}
I20260923 20:53:55.247273 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.243325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41975 (local address 127.0.97.1:53385)
0923 20:53:55.244498 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.244514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:55.244676 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:55.245016 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:55.245022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:55.245046 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:53:55.246077 (+ 1031us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:53:55.246079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:53:55.246092 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:53:55.246126 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:53:55.246441 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:53:55.246445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:53:55.246449 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 20:53:55.246612 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 20:53:55.246658 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":935,"threads_started":1}
I20260923 20:53:55.248267 5476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc503ab2b2904221812d6dbe41b1266b" candidate_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b4f643bb594521b971fd38aeafdcf1" is_pre_election: true
I20260923 20:53:55.248505 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc503ab2b2904221812d6dbe41b1266b" candidate_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" is_pre_election: true
I20260923 20:53:55.248798 5476 raft_consensus.cc:2454] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2cd6979e7c418cb84302aab7ff74d9 in term 0.
I20260923 20:53:55.249006 5346 raft_consensus.cc:2454] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f2cd6979e7c418cb84302aab7ff74d9 in term 0.
I20260923 20:53:55.249548 5149 leader_election.cc:302] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2cd6979e7c418cb84302aab7ff74d9, b5b4f643bb594521b971fd38aeafdcf1; no voters:
I20260923 20:53:55.250020 5547 raft_consensus.cc:2781] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:53:55.250195 5547 raft_consensus.cc:482] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:53:55.250303 5547 raft_consensus.cc:3037] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:55.253154 5547 raft_consensus.cc:504] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.253783 5547 leader_election.cc:288] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [CANDIDATE]: Term 1 election: Requested vote from peers b5b4f643bb594521b971fd38aeafdcf1 (127.0.97.3:33827), 0373c36eb5ff4d82b178b47f1f2bcf1c (127.0.97.2:41975)
I20260923 20:53:55.254443 5476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc503ab2b2904221812d6dbe41b1266b" candidate_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b4f643bb594521b971fd38aeafdcf1"
I20260923 20:53:55.254595 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc503ab2b2904221812d6dbe41b1266b" candidate_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c"
I20260923 20:53:55.254707 5476 raft_consensus.cc:3037] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:55.254819 5346 raft_consensus.cc:3037] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:55.258179 5476 raft_consensus.cc:2454] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2cd6979e7c418cb84302aab7ff74d9 in term 1.
I20260923 20:53:55.258179 5346 raft_consensus.cc:2454] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f2cd6979e7c418cb84302aab7ff74d9 in term 1.
I20260923 20:53:55.258711 5149 leader_election.cc:302] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f2cd6979e7c418cb84302aab7ff74d9, b5b4f643bb594521b971fd38aeafdcf1; no voters:
I20260923 20:53:55.259080 5547 raft_consensus.cc:2781] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:55.259605 5547 raft_consensus.cc:686] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 1 LEADER]: Becoming Leader. State: Replica: 9f2cd6979e7c418cb84302aab7ff74d9, State: Running, Role: LEADER
I20260923 20:53:55.260130 5547 consensus_queue.cc:237] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } }
I20260923 20:53:55.265130 5073 catalog_manager.cc:5495] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f2cd6979e7c418cb84302aab7ff74d9 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f2cd6979e7c418cb84302aab7ff74d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20260923 20:53:55.279935 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.277666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.0.97.1:39639)
0923 20:53:55.277886 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.277891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.278010 (+ 119us) server_negotiation.cc:407] Connection header received
0923 20:53:55.278259 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.278263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.278340 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.278460 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.279322 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.279327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.279334 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.279354 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.279385 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.279396 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.279515 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.279601 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.279661 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:53:55.288390 5346 raft_consensus.cc:1261] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 1 FOLLOWER]: Refusing update from remote peer 9f2cd6979e7c418cb84302aab7ff74d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:55.288391 5476 raft_consensus.cc:1261] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 1 FOLLOWER]: Refusing update from remote peer 9f2cd6979e7c418cb84302aab7ff74d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:53:55.289507 5550 consensus_queue.cc:1035] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b4f643bb594521b971fd38aeafdcf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:55.289886 5547 consensus_queue.cc:1035] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0373c36eb5ff4d82b178b47f1f2bcf1c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:53:55.304363 5553 mvcc.cc:205] Tried to move back new op lower bound from 7332646237330108416 to 7332646237229584384. Current Snapshot: MvccSnapshot[applied={T|T < 7332646237330108416}]
I20260923 20:53:55.320907 5074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:55.321185 5074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:53:55.324893 5074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11515010d05d4d9fafd75c423a77e39d: Sending DeleteTablet for 3 replicas of tablet dc503ab2b2904221812d6dbe41b1266b
I20260923 20:53:55.326241 5456 tablet_service.cc:1514] Processing DeleteTablet for tablet dc503ab2b2904221812d6dbe41b1266b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:55 UTC) from {username='slave'} at 127.0.0.1:60796
I20260923 20:53:55.326246 5195 tablet_service.cc:1514] Processing DeleteTablet for tablet dc503ab2b2904221812d6dbe41b1266b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:55 UTC) from {username='slave'} at 127.0.0.1:40036
I20260923 20:53:55.326368 5326 tablet_service.cc:1514] Processing DeleteTablet for tablet dc503ab2b2904221812d6dbe41b1266b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:53:55 UTC) from {username='slave'} at 127.0.0.1:57350
I20260923 20:53:55.329360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5133
I20260923 20:53:55.337462 5564 tablet_replica.cc:330] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: stopping tablet replica
I20260923 20:53:55.338001 5564 raft_consensus.cc:2229] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:53:55.338737 5564 raft_consensus.cc:2258] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:55.341089 5563 tablet_replica.cc:330] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: stopping tablet replica
I20260923 20:53:55.341596 5563 raft_consensus.cc:2229] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:55.341866 5565 tablet_replica.cc:330] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: stopping tablet replica
I20260923 20:53:55.342154 5563 raft_consensus.cc:2258] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:55.342350 5565 raft_consensus.cc:2229] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:53:55.343112 5565 raft_consensus.cc:2258] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:53:55.344695 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.331178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.0.97.1:39639)
0923 20:53:55.331443 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.331452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.341473 (+ 10021us) server_negotiation.cc:407] Connection header received
0923 20:53:55.341890 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.341899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.341965 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.342051 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.343726 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.343735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.343743 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.343768 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.343807 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.343824 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.343930 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.344289 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.344368 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:53:55.345031 5564 ts_tablet_manager.cc:1905] T dc503ab2b2904221812d6dbe41b1266b P 9f2cd6979e7c418cb84302aab7ff74d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:55.345986 5255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:55.346555 5565 ts_tablet_manager.cc:1905] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:55.347210 5563 ts_tablet_manager.cc:1905] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:53:55.354172 5563 ts_tablet_manager.cc:1918] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:55.354378 5563 log.cc:1196] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/dc503ab2b2904221812d6dbe41b1266b
I20260923 20:53:55.354552 5565 ts_tablet_manager.cc:1918] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:53:55.354766 5565 log.cc:1196] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/dc503ab2b2904221812d6dbe41b1266b
I20260923 20:53:55.355058 5563 ts_tablet_manager.cc:1939] T dc503ab2b2904221812d6dbe41b1266b P b5b4f643bb594521b971fd38aeafdcf1: Deleting consensus metadata
I20260923 20:53:55.355798 5565 ts_tablet_manager.cc:1939] T dc503ab2b2904221812d6dbe41b1266b P 0373c36eb5ff4d82b178b47f1f2bcf1c: Deleting consensus metadata
I20260923 20:53:55.357221 5059 catalog_manager.cc:4841] TS b5b4f643bb594521b971fd38aeafdcf1 (127.0.97.3:33827): tablet dc503ab2b2904221812d6dbe41b1266b (table test-table [id=aaaaff970e2a45b48f2d9c52c6a3479f]) successfully deleted
I20260923 20:53:55.357538 5060 catalog_manager.cc:4841] TS 0373c36eb5ff4d82b178b47f1f2bcf1c (127.0.97.2:41975): tablet dc503ab2b2904221812d6dbe41b1266b (table test-table [id=aaaaff970e2a45b48f2d9c52c6a3479f]) successfully deleted
I20260923 20:53:55.446316 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5133
I20260923 20:53:55.465082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5264
W20260923 20:53:55.465216 5058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39639 (error 108)
I20260923 20:53:55.467635 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.467047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40060)
0923 20:53:55.467279 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.467438 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:53:55.467906 5058 catalog_manager.cc:4584] TS 9f2cd6979e7c418cb84302aab7ff74d9 (127.0.97.1:39639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc503ab2b2904221812d6dbe41b1266b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
I20260923 20:53:55.468266 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.465925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.0.97.2:41975)
0923 20:53:55.466108 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.466114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.466200 (+ 86us) server_negotiation.cc:407] Connection header received
0923 20:53:55.466420 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.466425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.466522 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.466636 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.467603 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.467607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.467610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.467625 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.467656 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.467667 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.467822 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.467962 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.468024 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:53:55.468853 5386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:55.521633 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.521152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40076)
0923 20:53:55.521369 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.521479 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 20:53:55.522719 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.522373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40086)
0923 20:53:55.522525 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.522604 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 20:53:55.558053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5264
I20260923 20:53:55.575121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5394
I20260923 20:53:55.578032 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.575981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.97.3:33827)
0923 20:53:55.576170 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:55.576175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:55.576194 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:53:55.576349 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:55.576353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:55.576432 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:55.576520 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:55.577402 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:55.577406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:55.577409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:55.577422 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:55.577457 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:55.577469 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:55.577574 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:55.577634 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 20:53:55.577680 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:53:55.578723 5516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:55.583099 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.582595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40092)
0923 20:53:55.582788 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.582897 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:53:55.584168 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.583759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40094)
0923 20:53:55.583912 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.584001 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 20:53:55.665015 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.664526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40108)
0923 20:53:55.664749 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.664854 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 20:53:55.666175 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:55.665756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39639 (local address 127.0.0.1:40120)
0923 20:53:55.665991 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:55.666060 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260923 20:53:55.670025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5394
I20260923 20:53:55.687178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5043
I20260923 20:53:55.688321 5104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:55.793447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5043
2026-09-23T20:53:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2674 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 20:53:55.835529 388 mini_oidc.cc:152] Starting JWKS server
I20260923 20:53:55.838189 388 webserver.cc:460] Webserver started at https://127.0.0.1:46655/ using document root <none> and password file <none>
I20260923 20:53:55.838488 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:53:55.839138 388 webserver.cc:460] Webserver started at http://127.0.0.1:46293/ using document root <none> and password file <none>
2026-09-23T20:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:55Z Disabled control of system clock
I20260923 20:53:55.868168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35707
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35707
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/testchainca.cert with env {}
W20260923 20:53:56.230453 5577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:56.230804 5577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:56.230911 5577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:56.239367 5577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:56.239497 5577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:56.239539 5577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:56.239645 5577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:56.239702 5577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:53:56.239773 5577 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:53:56.239818 5577 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46655/jwks.json
I20260923 20:53:56.251624 5577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:56.253023 5577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:56.254379 5577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:56.266891 5585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:56.266991 5583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:56.267012 5582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:56.267688 5577 server_base.cc:1034] running on GCE node
I20260923 20:53:56.268548 5577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:56.270112 5577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:56.271353 5577 hybrid_clock.cc:648] HybridClock initialized: now 1790196836271317 us; error 41 us; skew 500 ppm
I20260923 20:53:56.271739 5577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:56.274219 5577 webserver.cc:460] Webserver started at http://127.0.97.62:42093/ using document root <none> and password file <none>
I20260923 20:53:56.274946 5577 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:56.275090 5577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:56.275413 5577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:56.278426 5577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "f131f34b43a046e08a9c8551cfdb86e0"
format_stamp: "Formatted at 2026-09-23 20:53:56 on dist-test-slave-plcg"
server_key: "cf9054c926c23727530373b5063918ec"
server_key_iv: "aad25aae3219dd2fad07b4932f225c22"
server_key_version: "encryptionkey@0"
I20260923 20:53:56.279254 5577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "f131f34b43a046e08a9c8551cfdb86e0"
format_stamp: "Formatted at 2026-09-23 20:53:56 on dist-test-slave-plcg"
server_key: "cf9054c926c23727530373b5063918ec"
server_key_iv: "aad25aae3219dd2fad07b4932f225c22"
server_key_version: "encryptionkey@0"
I20260923 20:53:56.284502 5577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:53:56.288259 5591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:56.289721 5577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:53:56.289958 5577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "f131f34b43a046e08a9c8551cfdb86e0"
format_stamp: "Formatted at 2026-09-23 20:53:56 on dist-test-slave-plcg"
server_key: "cf9054c926c23727530373b5063918ec"
server_key_iv: "aad25aae3219dd2fad07b4932f225c22"
server_key_version: "encryptionkey@0"
I20260923 20:53:56.290213 5577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:56.318897 5577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:56.319809 5577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:56.320132 5577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:56.339197 5577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35707
I20260923 20:53:56.339223 5642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35707 every 8 connection(s)
I20260923 20:53:56.340643 5577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:56.344838 5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:56.349201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5577
I20260923 20:53:56.349453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:56.350325 388 external_mini_cluster.cc:1428] Setting key e5ba7ee30ce81d0d7929599f2c1332c6
I20260923 20:53:56.358987 5643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0: Bootstrap starting.
I20260923 20:53:56.362298 5643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:56.363512 5643 log.cc:824] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:56.364964 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:56.352465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.97.62:35707)
0923 20:53:56.353532 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:56.353545 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:56.353590 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:53:56.355341 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:56.355377 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:56.355930 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:56.356396 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:56.357057 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:56.358438 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:56.359959 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:56.360706 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:56.360941 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:56.362264 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:56.362271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:56.362289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:56.362990 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:56.363092 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:56.363114 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:56.363253 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:56.363493 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 20:53:56.363851 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":373,"threads_started":1}
I20260923 20:53:56.367373 5643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0: No bootstrap required, opened a new log
I20260923 20:53:56.369164 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:53:56.376080 5643 raft_consensus.cc:348] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f131f34b43a046e08a9c8551cfdb86e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35707 } }
I20260923 20:53:56.376552 5643 raft_consensus.cc:374] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:56.376680 5643 raft_consensus.cc:729] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f131f34b43a046e08a9c8551cfdb86e0, State: Initialized, Role: FOLLOWER
I20260923 20:53:56.377321 5643 consensus_queue.cc:260] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f131f34b43a046e08a9c8551cfdb86e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35707 } }
I20260923 20:53:56.377589 5643 raft_consensus.cc:388] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:56.377739 5643 raft_consensus.cc:482] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:56.377997 5643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:56.380589 5643 raft_consensus.cc:504] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f131f34b43a046e08a9c8551cfdb86e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35707 } }
I20260923 20:53:56.381096 5643 leader_election.cc:302] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f131f34b43a046e08a9c8551cfdb86e0; no voters:
I20260923 20:53:56.381625 5643 leader_election.cc:288] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:56.381803 5648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:56.382597 5648 raft_consensus.cc:686] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [term 1 LEADER]: Becoming Leader. State: Replica: f131f34b43a046e08a9c8551cfdb86e0, State: Running, Role: LEADER
I20260923 20:53:56.383137 5648 consensus_queue.cc:237] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f131f34b43a046e08a9c8551cfdb86e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35707 } }
I20260923 20:53:56.383812 5643 sys_catalog.cc:564] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:56.386065 5650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f131f34b43a046e08a9c8551cfdb86e0. Latest consensus state: current_term: 1 leader_uuid: "f131f34b43a046e08a9c8551cfdb86e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f131f34b43a046e08a9c8551cfdb86e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35707 } } }
I20260923 20:53:56.386473 5650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:56.388937 5649 sys_catalog.cc:455] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f131f34b43a046e08a9c8551cfdb86e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f131f34b43a046e08a9c8551cfdb86e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35707 } } }
I20260923 20:53:56.389364 5649 sys_catalog.cc:458] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:56.390892 5656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:56.396454 5656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:56.408815 5656 catalog_manager.cc:1347] Generated new cluster ID: 41125ab6b036437fa0d58e448ea5df62
I20260923 20:53:56.408965 5656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:56.435321 5656 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:56.436765 5656 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:56.452503 5656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f131f34b43a046e08a9c8551cfdb86e0: Generated new TSK 0
I20260923 20:53:56.453644 5656 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:53:56.500156 5673 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 20:53:56.500355 5673 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:53:56.501268 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:56.479255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.97.62:35707)
0923 20:53:56.479493 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:56.479499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:56.479881 (+ 382us) server_negotiation.cc:407] Connection header received
0923 20:53:56.480215 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:56.480219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:56.480294 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:56.480426 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:53:56.481100 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:56.481875 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:56.484098 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:56.484736 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:56.484868 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:56.485565 (+ 697us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:53:56.500750 (+ 15185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:53:56.501053 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":86}
W20260923 20:53:56.501443 5646 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 20:53:56.503368 5672 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:53:56.479018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35707 (local address 127.0.0.1:56560)
0923 20:53:56.479758 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:56.479779 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:56.479978 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:56.480492 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:56.480499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:56.480513 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:53:56.480938 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:56.480950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:56.482026 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:56.482032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:56.483897 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:56.483915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:56.484894 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:56.484967 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:56.485167 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:56.485268 (+ 101us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:53:56.501080 (+ 15812us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:53:56.501482 (+ 402us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35707: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":356,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:35707: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 20:53:56.505522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5577
I20260923 20:53:56.506798 5638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:56.604592 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5577
2026-09-23T20:53:56Z chronyd exiting
I20260923 20:53:56.657946 388 test_util.cc:175] -----------------------------------------------
I20260923 20:53:56.658097 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790196790989944-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (827 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 20:53:56.661739 388 mini_oidc.cc:152] Starting JWKS server
I20260923 20:53:56.663727 388 webserver.cc:460] Webserver started at https://127.0.0.1:44443/ using document root <none> and password file <none>
I20260923 20:53:56.663946 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:53:56.664541 388 webserver.cc:460] Webserver started at http://127.0.0.1:35477/ using document root <none> and password file <none>
2026-09-23T20:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:56Z Disabled control of system clock
I20260923 20:53:56.691718 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36867
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36867
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/testchainca.cert with env {}
W20260923 20:53:57.022718 5680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:57.023051 5680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:57.023258 5680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:57.030725 5680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:57.030853 5680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:57.030951 5680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:57.031065 5680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:57.031147 5680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:53:57.031243 5680 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:53:57.031298 5680 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44443/jwks.json
I20260923 20:53:57.041592 5680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:57.043047 5680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:57.044399 5680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:57.056682 5685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:57.056742 5688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:57.056763 5686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:57.057296 5680 server_base.cc:1034] running on GCE node
I20260923 20:53:57.058290 5680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:57.059902 5680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:57.061146 5680 hybrid_clock.cc:648] HybridClock initialized: now 1790196837061082 us; error 59 us; skew 500 ppm
I20260923 20:53:57.061530 5680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:57.064019 5680 webserver.cc:460] Webserver started at http://127.0.97.62:39845/ using document root <none> and password file <none>
I20260923 20:53:57.064837 5680 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:57.064968 5680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:57.065304 5680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:57.068331 5680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "1800d79928e24c3183ccd19cf49eaa71"
format_stamp: "Formatted at 2026-09-23 20:53:57 on dist-test-slave-plcg"
server_key: "f9388d7189e42a4664341a0cf2f2ad06"
server_key_iv: "d5134ed0311f4cb0e991a787b7e291bb"
server_key_version: "encryptionkey@0"
I20260923 20:53:57.069193 5680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "1800d79928e24c3183ccd19cf49eaa71"
format_stamp: "Formatted at 2026-09-23 20:53:57 on dist-test-slave-plcg"
server_key: "f9388d7189e42a4664341a0cf2f2ad06"
server_key_iv: "d5134ed0311f4cb0e991a787b7e291bb"
server_key_version: "encryptionkey@0"
I20260923 20:53:57.074693 5680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:53:57.078383 5694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:57.079840 5680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:53:57.080049 5680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "1800d79928e24c3183ccd19cf49eaa71"
format_stamp: "Formatted at 2026-09-23 20:53:57 on dist-test-slave-plcg"
server_key: "f9388d7189e42a4664341a0cf2f2ad06"
server_key_iv: "d5134ed0311f4cb0e991a787b7e291bb"
server_key_version: "encryptionkey@0"
I20260923 20:53:57.080283 5680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:57.096987 5680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:57.097990 5680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:57.098290 5680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:57.117396 5680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36867
I20260923 20:53:57.117447 5745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36867 every 8 connection(s)
I20260923 20:53:57.118819 5680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:57.121533 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5680
I20260923 20:53:57.121747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:57.122632 388 external_mini_cluster.cc:1428] Setting key d312a75ba3ce006c4e1e3026d8d8872c
I20260923 20:53:57.123654 5746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:57.136966 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:57.124647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.0.97.62:36867)
0923 20:53:57.125676 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:57.125688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:57.125729 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:53:57.127296 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:57.127319 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:57.127820 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:57.128183 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:57.128751 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.130199 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.132103 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.132804 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.132945 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:57.134303 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:57.134312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:57.134333 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:57.135005 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:57.135100 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:57.135126 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:57.135242 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:57.135493 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 20:53:57.135926 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":394,"threads_started":1}
I20260923 20:53:57.137769 5746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71: Bootstrap starting.
I20260923 20:53:57.141235 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:53:57.141253 5746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:57.142529 5746 log.cc:824] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:57.145562 5746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71: No bootstrap required, opened a new log
I20260923 20:53:57.152910 5746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1800d79928e24c3183ccd19cf49eaa71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36867 } }
I20260923 20:53:57.153326 5746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:57.153409 5746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1800d79928e24c3183ccd19cf49eaa71, State: Initialized, Role: FOLLOWER
I20260923 20:53:57.154059 5746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1800d79928e24c3183ccd19cf49eaa71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36867 } }
I20260923 20:53:57.154271 5746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:57.154367 5746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:57.154497 5746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:57.156603 5746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1800d79928e24c3183ccd19cf49eaa71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36867 } }
I20260923 20:53:57.157117 5746 leader_election.cc:302] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1800d79928e24c3183ccd19cf49eaa71; no voters:
I20260923 20:53:57.157691 5746 leader_election.cc:288] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:57.157908 5751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:57.159140 5751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [term 1 LEADER]: Becoming Leader. State: Replica: 1800d79928e24c3183ccd19cf49eaa71, State: Running, Role: LEADER
I20260923 20:53:57.159699 5751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1800d79928e24c3183ccd19cf49eaa71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36867 } }
I20260923 20:53:57.160112 5746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:57.163043 5753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1800d79928e24c3183ccd19cf49eaa71. Latest consensus state: current_term: 1 leader_uuid: "1800d79928e24c3183ccd19cf49eaa71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1800d79928e24c3183ccd19cf49eaa71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36867 } } }
I20260923 20:53:57.163546 5753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:57.164115 5752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1800d79928e24c3183ccd19cf49eaa71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1800d79928e24c3183ccd19cf49eaa71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36867 } } }
I20260923 20:53:57.164446 5752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:57.169096 5759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:57.174685 5759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:57.184401 5759 catalog_manager.cc:1347] Generated new cluster ID: bd0eaa83077440ce8490c994f11926a1
I20260923 20:53:57.184557 5759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:57.209120 5759 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:57.210543 5759 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:57.222497 5759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1800d79928e24c3183ccd19cf49eaa71: Generated new TSK 0
I20260923 20:53:57.223443 5759 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:53:57.276081 5776 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 20:53:57.276226 5776 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:53:57.277405 5749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:57.249694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.0.97.62:36867)
0923 20:53:57.249934 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:57.249941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:57.250362 (+ 421us) server_negotiation.cc:407] Connection header received
0923 20:53:57.250635 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:57.250639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:57.250720 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:57.250904 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:53:57.251386 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.252152 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.253613 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.254173 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.254319 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:57.254873 (+ 554us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:53:57.276634 (+ 21761us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:53:57.277068 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:53:57.277549 5749 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 20:53:57.277607 5775 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:53:57.249544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36867 (local address 127.0.0.1:51862)
0923 20:53:57.250245 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:57.250266 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:57.250435 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:57.250928 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:57.250932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:57.250945 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:53:57.251218 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:57.251227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.252310 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.252317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:57.253445 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:57.253457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.254319 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.254370 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:57.254529 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:57.254565 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:53:57.276912 (+ 22347us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:53:57.277200 (+ 288us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":427,"threads_started":1}
I20260923 20:53:57.278925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5680
I20260923 20:53:57.280141 5741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:57.377132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5680
2026-09-23T20:53:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (771 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 20:53:57.433362 388 mini_oidc.cc:152] Starting JWKS server
I20260923 20:53:57.435424 388 webserver.cc:460] Webserver started at https://127.0.0.1:37825/ using document root <none> and password file <none>
I20260923 20:53:57.435705 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:53:57.436383 388 webserver.cc:460] Webserver started at http://127.0.0.1:39255/ using document root <none> and password file <none>
2026-09-23T20:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:57Z Disabled control of system clock
I20260923 20:53:57.464931 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42711
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42711
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37825/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:53:57.803503 5783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:57.803892 5783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:57.804068 5783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:57.811815 5783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:57.812005 5783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:57.812129 5783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:57.812247 5783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:57.812325 5783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:53:57.812430 5783 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:53:57.812492 5783 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37825/jwks.json
I20260923 20:53:57.823153 5783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37825/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:57.824673 5783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:57.826663 5783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:57.839361 5788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:57.839326 5791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:57.839423 5789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:57.840263 5783 server_base.cc:1034] running on GCE node
I20260923 20:53:57.841130 5783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:57.842810 5783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:57.844116 5783 hybrid_clock.cc:648] HybridClock initialized: now 1790196837844083 us; error 100 us; skew 500 ppm
I20260923 20:53:57.844525 5783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:57.847980 5783 webserver.cc:460] Webserver started at http://127.0.97.62:43405/ using document root <none> and password file <none>
I20260923 20:53:57.848868 5783 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:57.849018 5783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:57.849377 5783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:57.852603 5783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "e78dc79f1b8e4e18916d050f57d88f4b"
format_stamp: "Formatted at 2026-09-23 20:53:57 on dist-test-slave-plcg"
server_key: "1e9b1bf84459dd259245da7d1096cfaa"
server_key_iv: "da4f6367643bf61a1413f9d498919fe9"
server_key_version: "encryptionkey@0"
I20260923 20:53:57.853503 5783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "e78dc79f1b8e4e18916d050f57d88f4b"
format_stamp: "Formatted at 2026-09-23 20:53:57 on dist-test-slave-plcg"
server_key: "1e9b1bf84459dd259245da7d1096cfaa"
server_key_iv: "da4f6367643bf61a1413f9d498919fe9"
server_key_version: "encryptionkey@0"
I20260923 20:53:57.859427 5783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:53:57.863314 5797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:57.864866 5783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:53:57.865089 5783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "e78dc79f1b8e4e18916d050f57d88f4b"
format_stamp: "Formatted at 2026-09-23 20:53:57 on dist-test-slave-plcg"
server_key: "1e9b1bf84459dd259245da7d1096cfaa"
server_key_iv: "da4f6367643bf61a1413f9d498919fe9"
server_key_version: "encryptionkey@0"
I20260923 20:53:57.865346 5783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:57.887531 5783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:57.889002 5783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:57.889367 5783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:57.909454 5783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42711
I20260923 20:53:57.909472 5848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42711 every 8 connection(s)
I20260923 20:53:57.910962 5783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:57.915094 5849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:57.916746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5783
I20260923 20:53:57.916970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:57.917784 388 external_mini_cluster.cc:1428] Setting key 34b131d26e73f70fb86ff0573abce580
I20260923 20:53:57.929337 5849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b: Bootstrap starting.
I20260923 20:53:57.933020 5849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:57.933286 5852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:57.919942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.0.97.62:42711)
0923 20:53:57.921119 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:57.921133 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:57.921185 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:53:57.922881 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:57.922912 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:57.923495 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:57.923946 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:57.924574 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.926272 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.927960 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:57.928843 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:57.929109 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:57.930667 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:57.930678 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:57.930695 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:57.931303 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:57.931392 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:57.931415 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:57.931588 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:57.931854 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:53:57.932241 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":426,"threads_started":1}
I20260923 20:53:57.934494 5849 log.cc:824] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:57.937676 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:53:57.938045 5849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b: No bootstrap required, opened a new log
I20260923 20:53:57.946648 5849 raft_consensus.cc:348] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42711 } }
I20260923 20:53:57.947082 5849 raft_consensus.cc:374] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:57.947193 5849 raft_consensus.cc:729] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78dc79f1b8e4e18916d050f57d88f4b, State: Initialized, Role: FOLLOWER
I20260923 20:53:57.947808 5849 consensus_queue.cc:260] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42711 } }
I20260923 20:53:57.948071 5849 raft_consensus.cc:388] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:57.948228 5849 raft_consensus.cc:482] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:57.948412 5849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:57.951210 5849 raft_consensus.cc:504] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42711 } }
I20260923 20:53:57.951850 5849 leader_election.cc:302] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e78dc79f1b8e4e18916d050f57d88f4b; no voters:
I20260923 20:53:57.952490 5849 leader_election.cc:288] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:57.952749 5854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:57.953516 5854 raft_consensus.cc:686] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [term 1 LEADER]: Becoming Leader. State: Replica: e78dc79f1b8e4e18916d050f57d88f4b, State: Running, Role: LEADER
I20260923 20:53:57.954105 5854 consensus_queue.cc:237] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42711 } }
I20260923 20:53:57.954648 5849 sys_catalog.cc:564] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:57.956938 5856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e78dc79f1b8e4e18916d050f57d88f4b. Latest consensus state: current_term: 1 leader_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42711 } } }
I20260923 20:53:57.957335 5856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:57.957904 5855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78dc79f1b8e4e18916d050f57d88f4b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42711 } } }
I20260923 20:53:57.958222 5855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:57.960162 5860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:57.966042 5860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:57.975819 5860 catalog_manager.cc:1347] Generated new cluster ID: 4baaa5f62e0042e48e5ee2350d040fc7
I20260923 20:53:57.975962 5860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:57.994238 5860 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:57.995692 5860 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:58.006636 5860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e78dc79f1b8e4e18916d050f57d88f4b: Generated new TSK 0
I20260923 20:53:58.007596 5860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:53:58.031796 5852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:58.022782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.0.97.62:42711)
0923 20:53:58.023012 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:58.023019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:58.023581 (+ 562us) server_negotiation.cc:407] Connection header received
0923 20:53:58.023798 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:58.023805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:58.023913 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:58.024093 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:53:58.024558 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.025745 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.027202 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.027891 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.028036 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:58.028602 (+ 566us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:53:58.031093 (+ 2491us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:53:58.031455 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:53:58.031934 5852 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 20:53:58.033622 5878 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:53:58.022770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42711 (local address 127.0.0.1:40668)
0923 20:53:58.023469 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:58.023485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:58.023611 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:58.024111 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:58.024115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:58.024124 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:53:58.024407 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:58.024415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.025943 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.025947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:58.027033 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:58.027045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.028037 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.028108 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:58.028262 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:58.028298 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:53:58.031326 (+ 3028us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:53:58.033157 (+ 1831us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":1496,"thread_start_us":403,"threads_started":1}
I20260923 20:53:58.034739 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5783
I20260923 20:53:58.035953 5844 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 20:53:58.044119 5879 webserver.cc:533] Webserver: sslize failed:
E20260923 20:53:58.044270 5879 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:53:58.128546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5783
2026-09-23T20:53:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T20:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:53:58Z Disabled control of system clock
I20260923 20:53:58.213929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43639
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43639
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/testchainca.cert with env {}
W20260923 20:53:58.555402 5884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:53:58.555723 5884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:53:58.555812 5884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:53:58.563354 5884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:53:58.563470 5884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:53:58.563506 5884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:53:58.563556 5884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:53:58.563602 5884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:53:58.563665 5884 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:53:58.563705 5884 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 20:53:58.574132 5884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:53:58.575541 5884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:53:58.576831 5884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:53:58.590370 5889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:58.590539 5892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:53:58.590387 5890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:53:58.591251 5884 server_base.cc:1034] running on GCE node
I20260923 20:53:58.592192 5884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:53:58.593781 5884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:53:58.595075 5884 hybrid_clock.cc:648] HybridClock initialized: now 1790196838595010 us; error 60 us; skew 500 ppm
I20260923 20:53:58.595479 5884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:53:58.603538 5884 webserver.cc:460] Webserver started at http://127.0.97.62:42973/ using document root <none> and password file <none>
I20260923 20:53:58.604430 5884 fs_manager.cc:359] Metadata directory not provided
I20260923 20:53:58.604589 5884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:53:58.604939 5884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:53:58.608086 5884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "fc8a1175e6ef41dcb4c71bda64802041"
format_stamp: "Formatted at 2026-09-23 20:53:58 on dist-test-slave-plcg"
server_key: "76e4a1aac1650469b68867a52ec5444e"
server_key_iv: "584d8d7484b4e43424ca410da8ee2059"
server_key_version: "encryptionkey@0"
I20260923 20:53:58.608974 5884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc8a1175e6ef41dcb4c71bda64802041"
format_stamp: "Formatted at 2026-09-23 20:53:58 on dist-test-slave-plcg"
server_key: "76e4a1aac1650469b68867a52ec5444e"
server_key_iv: "584d8d7484b4e43424ca410da8ee2059"
server_key_version: "encryptionkey@0"
I20260923 20:53:58.614675 5884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:53:58.618953 5898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:53:58.620432 5884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:53:58.620657 5884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "fc8a1175e6ef41dcb4c71bda64802041"
format_stamp: "Formatted at 2026-09-23 20:53:58 on dist-test-slave-plcg"
server_key: "76e4a1aac1650469b68867a52ec5444e"
server_key_iv: "584d8d7484b4e43424ca410da8ee2059"
server_key_version: "encryptionkey@0"
I20260923 20:53:58.620918 5884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:53:58.639520 5884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:53:58.640450 5884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:53:58.640784 5884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:53:58.660254 5884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43639
I20260923 20:53:58.660300 5949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43639 every 8 connection(s)
I20260923 20:53:58.661715 5884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:53:58.665336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 5884
I20260923 20:53:58.665598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:53:58.665896 5950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:53:58.666483 388 external_mini_cluster.cc:1428] Setting key 5cce8b80eb4f2e439ca24d8f04ef6e64
I20260923 20:53:58.680061 5950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041: Bootstrap starting.
I20260923 20:53:58.682386 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:58.668524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.0.97.62:43639)
0923 20:53:58.669748 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:58.669761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:58.669808 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:53:58.672084 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:58.672112 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:58.672704 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:58.673079 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:53:58.673656 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.675285 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.677239 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.678116 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.678294 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:58.679597 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:53:58.679605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:53:58.679622 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:53:58.680301 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:53:58.680400 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:53:58.680425 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:53:58.680522 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:53:58.680796 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 20:53:58.681197 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":459,"threads_started":1}
I20260923 20:53:58.683902 5950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041: Neither blocks nor log segments found. Creating new log.
I20260923 20:53:58.685074 5950 log.cc:824] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041: Log is configured to *not* fsync() on all Append() calls
I20260923 20:53:58.686875 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:53:58.688364 5950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041: No bootstrap required, opened a new log
I20260923 20:53:58.695961 5950 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a1175e6ef41dcb4c71bda64802041" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260923 20:53:58.696499 5950 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:53:58.696610 5950 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8a1175e6ef41dcb4c71bda64802041, State: Initialized, Role: FOLLOWER
I20260923 20:53:58.697333 5950 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a1175e6ef41dcb4c71bda64802041" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260923 20:53:58.697559 5950 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:53:58.697657 5950 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:53:58.697794 5950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:53:58.700008 5950 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a1175e6ef41dcb4c71bda64802041" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260923 20:53:58.700512 5950 leader_election.cc:302] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8a1175e6ef41dcb4c71bda64802041; no voters:
I20260923 20:53:58.701118 5950 leader_election.cc:288] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:53:58.701321 5955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:53:58.702133 5955 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [term 1 LEADER]: Becoming Leader. State: Replica: fc8a1175e6ef41dcb4c71bda64802041, State: Running, Role: LEADER
I20260923 20:53:58.702692 5955 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a1175e6ef41dcb4c71bda64802041" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } }
I20260923 20:53:58.703372 5950 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:53:58.707484 5956 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8a1175e6ef41dcb4c71bda64802041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a1175e6ef41dcb4c71bda64802041" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } } }
I20260923 20:53:58.707418 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8a1175e6ef41dcb4c71bda64802041. Latest consensus state: current_term: 1 leader_uuid: "fc8a1175e6ef41dcb4c71bda64802041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8a1175e6ef41dcb4c71bda64802041" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43639 } } }
I20260923 20:53:58.707902 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:58.707902 5956 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041 [sys.catalog]: This master's current role is: LEADER
I20260923 20:53:58.711022 5964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:53:58.717170 5964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:53:58.726960 5964 catalog_manager.cc:1347] Generated new cluster ID: 113299218a1f45728bb426a2603a4bb0
I20260923 20:53:58.727103 5964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:53:58.751104 5964 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:53:58.752542 5964 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:53:58.760921 5964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8a1175e6ef41dcb4c71bda64802041: Generated new TSK 0
I20260923 20:53:58.761904 5964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:53:58.856801 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:53:58.783506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.0.97.62:43639)
0923 20:53:58.783702 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:53:58.783709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:53:58.784031 (+ 322us) server_negotiation.cc:407] Connection header received
0923 20:53:58.784353 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:53:58.784360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:53:58.784463 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:53:58.784630 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:53:58.785328 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.786211 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.787622 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.788088 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.788224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:58.788753 (+ 529us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:53:58.856013 (+ 67260us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:53:58.856484 (+ 471us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55200: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 20:53:58.856964 5953 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55200: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 20:53:58.857084 5979 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:53:58.783292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43639 (local address 127.0.0.1:55200)
0923 20:53:58.783921 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0923 20:53:58.783938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:53:58.784090 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:53:58.784660 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:53:58.784665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:53:58.784705 (+ 40us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:53:58.785134 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:58.785161 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.786362 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.786367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:58.787470 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:53:58.787481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:53:58.788211 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:53:58.788280 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:53:58.788440 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:53:58.788476 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:53:58.856360 (+ 67884us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:53:58.856642 (+ 282us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":340,"threads_started":1}
I20260923 20:53:58.858217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 5884
I20260923 20:53:58.859543 5945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:53:58.958611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 5884
2026-09-23T20:53:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (808 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (421 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (441 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:53:59 dist-test-slave-plcg krb5kdc[5994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:53:59 dist-test-slave-plcg krb5kdc[5994](info): set up 2 sockets
Sep 23 20:53:59 dist-test-slave-plcg krb5kdc[5994](info): commencing operation
krb5kdc: starting...
W20260923 20:54:00.918584 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:54:01 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:54:01.138391 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38793
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:01.477833 6007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:01.478204 6007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:01.478363 6007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:01.485926 6007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:01.486092 6007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:01.486202 6007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:01.486310 6007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:01.486397 6007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:01.496881 6007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:01.498440 6007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:01.499809 6007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:01.512729 6012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:01.512795 6015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:01.512804 6013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:01.513605 6007 server_base.cc:1034] running on GCE node
I20260923 20:54:01.514582 6007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:01.516182 6007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:01.517480 6007 hybrid_clock.cc:648] HybridClock initialized: now 1790196841517441 us; error 63 us; skew 500 ppm
I20260923 20:54:01.517951 6007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:01 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:01.521384 6007 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:54:01.523144 6007 webserver.cc:460] Webserver started at http://127.0.97.62:44841/ using document root <none> and password file <none>
I20260923 20:54:01.523911 6007 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:01.524071 6007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:01.524411 6007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:01.527609 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "d1c3b0b1a26e46728b0091de019abebe"
format_stamp: "Formatted at 2026-09-23 20:54:01 on dist-test-slave-plcg"
server_key: "98edc0576cb7021a2b964bf6416918f6"
server_key_iv: "dd4f66a9c34eaf09c0ac74b301b6e517"
server_key_version: "encryptionkey@0"
I20260923 20:54:01.528469 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1c3b0b1a26e46728b0091de019abebe"
format_stamp: "Formatted at 2026-09-23 20:54:01 on dist-test-slave-plcg"
server_key: "98edc0576cb7021a2b964bf6416918f6"
server_key_iv: "dd4f66a9c34eaf09c0ac74b301b6e517"
server_key_version: "encryptionkey@0"
I20260923 20:54:01.534206 6007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:54:01.538100 6022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:01.539685 6007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:54:01.539945 6007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "d1c3b0b1a26e46728b0091de019abebe"
format_stamp: "Formatted at 2026-09-23 20:54:01 on dist-test-slave-plcg"
server_key: "98edc0576cb7021a2b964bf6416918f6"
server_key_iv: "dd4f66a9c34eaf09c0ac74b301b6e517"
server_key_version: "encryptionkey@0"
I20260923 20:54:01.540198 6007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:01.556361 6007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:01.561203 6007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:01.561537 6007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:01.581212 6007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38793
I20260923 20:54:01.581241 6073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38793 every 8 connection(s)
I20260923 20:54:01.582885 6007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:01.587131 6074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:01.590044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6007
I20260923 20:54:01.590260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:54:01.591048 388 external_mini_cluster.cc:1428] Setting key b2c7ea7d469d283001bc61dc6b4332dc
I20260923 20:54:01.600629 6074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe: Bootstrap starting.
I20260923 20:54:01.604318 6074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:01.605640 6074 log.cc:824] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:54:01 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:01.609514 6074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe: No bootstrap required, opened a new log
I20260923 20:54:01.618571 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:01.593451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.0.97.62:38793)
0923 20:54:01.594363 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:01.594377 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:01.594437 (+ 60us) server_negotiation.cc:407] Connection header received
0923 20:54:01.596242 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:01.596268 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:01.596727 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:01.597218 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:01.599059 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:01.600461 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:01.601932 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:01.602970 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:01.603309 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:01.608174 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:01.608183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:01.608201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:01.608348 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:01.613068 (+ 4720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:01.614183 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:01.614192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:01.614200 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:01.614441 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:01.614945 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:01.614950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:01.614952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:01.615633 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:01.615922 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:01.616576 (+ 654us) server_negotiation.cc:299] Negotiation successful
0923 20:54:01.617174 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":401,"threads_started":1}
I20260923 20:54:01.619660 6074 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b0b1a26e46728b0091de019abebe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38793 } }
I20260923 20:54:01.620110 6074 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:01.620334 6074 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c3b0b1a26e46728b0091de019abebe, State: Initialized, Role: FOLLOWER
I20260923 20:54:01.621052 6074 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b0b1a26e46728b0091de019abebe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38793 } }
I20260923 20:54:01.621364 6074 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:01.621531 6074 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:01.621743 6074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:01.624202 6074 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b0b1a26e46728b0091de019abebe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38793 } }
I20260923 20:54:01.624819 6074 leader_election.cc:302] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c3b0b1a26e46728b0091de019abebe; no voters:
I20260923 20:54:01.625640 6074 leader_election.cc:288] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:54:01.625679 6079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:01.627789 6079 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [term 1 LEADER]: Becoming Leader. State: Replica: d1c3b0b1a26e46728b0091de019abebe, State: Running, Role: LEADER
I20260923 20:54:01.628765 6074 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:01.628765 6079 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b0b1a26e46728b0091de019abebe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38793 } }
I20260923 20:54:01.634078 6080 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1c3b0b1a26e46728b0091de019abebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b0b1a26e46728b0091de019abebe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38793 } } }
I20260923 20:54:01.634054 6081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c3b0b1a26e46728b0091de019abebe. Latest consensus state: current_term: 1 leader_uuid: "d1c3b0b1a26e46728b0091de019abebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b0b1a26e46728b0091de019abebe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38793 } } }
I20260923 20:54:01.634563 6081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:01.634562 6080 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:01.635615 6090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:01.643662 6090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:01.661989 6090 catalog_manager.cc:1347] Generated new cluster ID: 0a4b1704d756476ca47f01d45e8bf7cc
I20260923 20:54:01.662153 6090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:01.686766 6090 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:54:01.688293 6090 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:01.697978 6090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1c3b0b1a26e46728b0091de019abebe: Generated new TSK 0
I20260923 20:54:01.699000 6090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:54:01.766676 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38793
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:02.126457 6100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:02.126839 6100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:02.127012 6100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:02.135058 6100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:02.135228 6100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:02.135411 6100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:02.145699 6100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:02.147117 6100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:02.148481 6100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:02.162950 6105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:02.163033 6108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:02.163004 6106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:02.163892 6100 server_base.cc:1034] running on GCE node
I20260923 20:54:02.164489 6100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:02.165460 6100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:02.166698 6100 hybrid_clock.cc:648] HybridClock initialized: now 1790196842166650 us; error 43 us; skew 500 ppm
I20260923 20:54:02.167136 6100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:02 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:02.171036 6100 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:54:02.172693 6100 webserver.cc:460] Webserver started at http://127.0.97.1:43093/ using document root <none> and password file <none>
I20260923 20:54:02.173434 6100 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:02.173595 6100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:02.173997 6100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:02.177652 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "284ffbfd3f6e472fae851228fbba9309"
format_stamp: "Formatted at 2026-09-23 20:54:02 on dist-test-slave-plcg"
server_key: "f9cb1d6548045da08e85d1080ded2ed8"
server_key_iv: "32f2abf5e47c18e63b33b3467e6bce35"
server_key_version: "encryptionkey@0"
I20260923 20:54:02.178692 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "284ffbfd3f6e472fae851228fbba9309"
format_stamp: "Formatted at 2026-09-23 20:54:02 on dist-test-slave-plcg"
server_key: "f9cb1d6548045da08e85d1080ded2ed8"
server_key_iv: "32f2abf5e47c18e63b33b3467e6bce35"
server_key_version: "encryptionkey@0"
I20260923 20:54:02.184399 6100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:54:02.189637 6115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:02.191277 6100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:54:02.191510 6100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "284ffbfd3f6e472fae851228fbba9309"
format_stamp: "Formatted at 2026-09-23 20:54:02 on dist-test-slave-plcg"
server_key: "f9cb1d6548045da08e85d1080ded2ed8"
server_key_iv: "32f2abf5e47c18e63b33b3467e6bce35"
server_key_version: "encryptionkey@0"
I20260923 20:54:02.191764 6100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:02.209371 6100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:02.213682 6100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:02.214046 6100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:02.215090 6100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:02.216672 6100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:02.216748 6100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:02.216912 6100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:02.216964 6100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:02.252274 6100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33863
I20260923 20:54:02.252341 6227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33863 every 8 connection(s)
I20260923 20:54:02.253640 6100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:02.261523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6100
I20260923 20:54:02.261786 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:54:02.262463 388 external_mini_cluster.cc:1428] Setting key d3e1374f622e778aa4affb2227c704f2
Sep 23 20:54:02 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:02.275854 6230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:02.256332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55191 (local address 127.0.97.62:38793)
0923 20:54:02.256780 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:02.256785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:02.257749 (+ 964us) server_negotiation.cc:407] Connection header received
0923 20:54:02.259668 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:02.259673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:02.259779 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:02.259906 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:02.261748 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.262652 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.264349 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.264781 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.264928 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:02.269637 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:02.269644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:02.269650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:02.269687 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:02.272490 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:02.273354 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:02.273360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:02.273369 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:02.273488 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:02.274015 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:02.274025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:02.274027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:02.274295 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:02.274458 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:02.275267 (+ 809us) server_negotiation.cc:299] Negotiation successful
0923 20:54:02.275496 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":252,"threads_started":1}
I20260923 20:54:02.276901 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:02.256833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38793 (local address 127.0.97.1:55191)
0923 20:54:02.257574 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:02.257616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:02.259365 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:02.260158 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:02.260168 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:02.260698 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:02.261478 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:02.261509 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.262892 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.262896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:02.264169 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:02.264182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.265052 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.265060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:02.265318 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:02.266202 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:02.266237 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:02.269332 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:02.272694 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:02.272705 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:02.272727 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:02.273208 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:02.273610 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:02.273615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:02.273619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:02.273883 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:02.274434 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:02.274443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:02.274836 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 20:54:02.275141 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 20:54:02.275717 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":305,"threads_started":1}
I20260923 20:54:02.278754 6228 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38793
I20260923 20:54:02.279155 6228 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:02.280175 6228 heartbeater.cc:511] Master 127.0.97.62:38793 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:54:02.282779 6039 ts_manager.cc:194] Registered new tserver with Master: 284ffbfd3f6e472fae851228fbba9309 (127.0.97.1:33863)
I20260923 20:54:02.284368 6039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:54:02.309934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38793
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:02.653854 6234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:02.654168 6234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:02.654269 6234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:02.662081 6234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:02.662220 6234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:02.662338 6234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:02.673214 6234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:02.674671 6234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:02.676010 6234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:02.690811 6242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:02.690971 6239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:02.690937 6240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:02.692044 6234 server_base.cc:1034] running on GCE node
I20260923 20:54:02.692749 6234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:02.693763 6234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:02.695063 6234 hybrid_clock.cc:648] HybridClock initialized: now 1790196842694992 us; error 62 us; skew 500 ppm
I20260923 20:54:02.695444 6234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:02 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:02.699164 6234 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:54:02.700850 6234 webserver.cc:460] Webserver started at http://127.0.97.2:40549/ using document root <none> and password file <none>
I20260923 20:54:02.701550 6234 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:02.701659 6234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:02.702028 6234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:02.705046 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "1a637da8240141879b494ab5468e2b7d"
format_stamp: "Formatted at 2026-09-23 20:54:02 on dist-test-slave-plcg"
server_key: "322b4966b6a728f0172c14c4abea691d"
server_key_iv: "99c32235b07af4a0b8c6c5ad97ed005d"
server_key_version: "encryptionkey@0"
I20260923 20:54:02.705981 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1a637da8240141879b494ab5468e2b7d"
format_stamp: "Formatted at 2026-09-23 20:54:02 on dist-test-slave-plcg"
server_key: "322b4966b6a728f0172c14c4abea691d"
server_key_iv: "99c32235b07af4a0b8c6c5ad97ed005d"
server_key_version: "encryptionkey@0"
I20260923 20:54:02.711526 6234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:54:02.715328 6249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:02.716871 6234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:54:02.717079 6234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "1a637da8240141879b494ab5468e2b7d"
format_stamp: "Formatted at 2026-09-23 20:54:02 on dist-test-slave-plcg"
server_key: "322b4966b6a728f0172c14c4abea691d"
server_key_iv: "99c32235b07af4a0b8c6c5ad97ed005d"
server_key_version: "encryptionkey@0"
I20260923 20:54:02.717332 6234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:02.734633 6234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:02.738858 6234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:02.739181 6234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:02.740234 6234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:02.741729 6234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:02.741842 6234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:02.741989 6234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:02.742043 6234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:02.776937 6234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35069
I20260923 20:54:02.777026 6361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35069 every 8 connection(s)
I20260923 20:54:02.778349 6234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:02.782195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6234
I20260923 20:54:02.782382 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:54:02.782879 388 external_mini_cluster.cc:1428] Setting key 1801634c9c8d02da3d063eee81c04337
Sep 23 20:54:02 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:02.800933 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:02.780917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36691 (local address 127.0.97.62:38793)
0923 20:54:02.781354 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:02.781361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:02.782555 (+ 1194us) server_negotiation.cc:407] Connection header received
0923 20:54:02.784830 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:02.784834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:02.784897 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:02.785032 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:02.786821 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.787978 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.789646 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.790430 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.790578 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:02.794600 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:02.794605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:02.794609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:02.794644 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:02.797482 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:02.798448 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:02.798460 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:02.798463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:02.798595 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:02.799092 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:02.799102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:02.799105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:02.799388 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:02.799552 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:02.800326 (+ 774us) server_negotiation.cc:299] Negotiation successful
0923 20:54:02.800556 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":206,"threads_started":1}
I20260923 20:54:02.802058 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:02.781516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38793 (local address 127.0.97.2:36691)
0923 20:54:02.782366 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:02.782416 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:02.784462 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:02.785256 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:02.785265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:02.785795 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:02.786618 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:02.786638 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.788146 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.788149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:02.789442 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:02.789456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:02.790675 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:02.790680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:02.790904 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:02.791676 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:02.791702 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:02.794312 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:02.797673 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:02.797685 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:02.797705 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:02.798294 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:02.798722 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:02.798727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:02.798731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:02.798955 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:02.799528 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:02.799536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:02.799929 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 20:54:02.800213 (+ 284us) client_negotiation.cc:240] Negotiation successful
0923 20:54:02.800775 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":381,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:54:02.803635 6362 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38793
I20260923 20:54:02.803996 6362 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:02.804859 6362 heartbeater.cc:511] Master 127.0.97.62:38793 requested a full tablet report, sending...
I20260923 20:54:02.806522 6039 ts_manager.cc:194] Registered new tserver with Master: 1a637da8240141879b494ab5468e2b7d (127.0.97.2:35069)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 20:54:02.807471 6039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:54:02.834586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38793
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:03.175352 6368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:03.175724 6368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:03.175910 6368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:03.183398 6368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:03.183600 6368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:03.183800 6368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:03.195741 6368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:03.197154 6368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:03.198606 6368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:03.212558 6373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:03.212603 6376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:03.212626 6374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:03.213961 6368 server_base.cc:1034] running on GCE node
I20260923 20:54:03.214675 6368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:03.215744 6368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:03.216969 6368 hybrid_clock.cc:648] HybridClock initialized: now 1790196843216929 us; error 41 us; skew 500 ppm
I20260923 20:54:03.217321 6368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.220868 6368 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:54:03.222527 6368 webserver.cc:460] Webserver started at http://127.0.97.3:36825/ using document root <none> and password file <none>
I20260923 20:54:03.223268 6368 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:03.223416 6368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:03.223744 6368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:03.226961 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "f7a2a3ef50164f72b4b39aa2d58fa42b"
format_stamp: "Formatted at 2026-09-23 20:54:03 on dist-test-slave-plcg"
server_key: "4e4693e86057665abbb2527b95b02b61"
server_key_iv: "bf4384b913f60db4065bf2553195a31f"
server_key_version: "encryptionkey@0"
I20260923 20:54:03.227834 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f7a2a3ef50164f72b4b39aa2d58fa42b"
format_stamp: "Formatted at 2026-09-23 20:54:03 on dist-test-slave-plcg"
server_key: "4e4693e86057665abbb2527b95b02b61"
server_key_iv: "bf4384b913f60db4065bf2553195a31f"
server_key_version: "encryptionkey@0"
I20260923 20:54:03.234015 6368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:54:03.237994 6383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:03.239519 6368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:54:03.239743 6368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "f7a2a3ef50164f72b4b39aa2d58fa42b"
format_stamp: "Formatted at 2026-09-23 20:54:03 on dist-test-slave-plcg"
server_key: "4e4693e86057665abbb2527b95b02b61"
server_key_iv: "bf4384b913f60db4065bf2553195a31f"
server_key_version: "encryptionkey@0"
I20260923 20:54:03.239995 6368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:03.268579 6368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:03.273339 6368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:03.273708 6368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:03.274775 6368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:03.276232 6368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:03.276382 6368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:03.276541 6368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:03.276618 6368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:03.287743 6228 heartbeater.cc:503] Master 127.0.97.62:38793 was elected leader, sending a full tablet report...
I20260923 20:54:03.312388 6368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43543
I20260923 20:54:03.312441 6495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43543 every 8 connection(s)
I20260923 20:54:03.313788 6368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:03.316644 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6368
I20260923 20:54:03.316833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:54:03.317272 388 external_mini_cluster.cc:1428] Setting key 646cb9c24a7d4c7091987851bf9a014b
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:03.335220 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.317942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36705 (local address 127.0.97.62:38793)
0923 20:54:03.318793 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.318800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.318821 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:54:03.320182 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.320187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.320270 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.320413 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.322251 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.323034 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.324704 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.325109 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.325257 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.328854 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:03.328859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:03.328863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:03.328898 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:03.331717 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.332576 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.332583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.332588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.332717 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.333219 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.333226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.333230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.333544 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:03.333718 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:03.334575 (+ 857us) server_negotiation.cc:299] Negotiation successful
0923 20:54:03.334813 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":601,"threads_started":1}
I20260923 20:54:03.336216 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.317107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38793 (local address 127.0.97.3:36705)
0923 20:54:03.318060 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:03.318115 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:03.319881 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:03.320654 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:03.320663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:03.321346 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:03.322017 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:03.322038 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.323235 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.323245 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:03.324519 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:03.324537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.325357 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.325364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:03.325612 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.326331 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:03.326356 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:03.328593 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:03.331908 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:03.331917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:03.331939 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:03.332401 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:03.332843 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:03.332848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:03.332852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:03.333085 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:03.333710 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:03.333719 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:03.334184 (+ 465us) client_negotiation.cc:769] Sending connection context
0923 20:54:03.334465 (+ 281us) client_negotiation.cc:240] Negotiation successful
0923 20:54:03.335065 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":393,"threads_started":1}
I20260923 20:54:03.337773 6496 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38793
I20260923 20:54:03.338146 6496 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:03.338848 6496 heartbeater.cc:511] Master 127.0.97.62:38793 requested a full tablet report, sending...
I20260923 20:54:03.340421 6039 ts_manager.cc:194] Registered new tserver with Master: f7a2a3ef50164f72b4b39aa2d58fa42b (127.0.97.3:43543)
I20260923 20:54:03.341286 6039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36705
I20260923 20:54:03.345295 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/test-admin.keytab.
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.371388 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 20:54:03.376664 6504 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:03.374249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.0.97.1:33863)
0923 20:54:03.374919 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.374926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.374954 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:54:03.375016 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.375023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.375235 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.375370 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.376265 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59020: BlockingRecv error: recv got EOF from 127.0.0.1:59020 (error 108)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":358,"threads_started":1}
W20260923 20:54:03.377456 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.379343 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.380872 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:54:03.397267 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.382937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.97.1:33863)
0923 20:54:03.383151 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.383158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.383297 (+ 139us) server_negotiation.cc:407] Connection header received
0923 20:54:03.383472 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.383476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.383535 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.383619 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.384649 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.385625 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.386986 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.387572 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.387725 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.391625 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:03.391638 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:03.391655 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:03.391690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:03.394515 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.395297 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.395303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.395306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.395433 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.395800 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.395803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.395805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.396145 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:03.396471 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:03.396805 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:54:03.397007 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260923 20:54:03.403357 6504 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:03.401492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.0.97.1:33863)
0923 20:54:03.401744 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.401750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.402011 (+ 261us) server_negotiation.cc:407] Connection header received
0923 20:54:03.402149 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.402153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.402223 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.402305 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.403032 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59030: BlockingRecv error: recv got EOF from 127.0.0.1:59030 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260923 20:54:03.403550 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.405084 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.406661 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:54:03.420899 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.408086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.0.97.1:33863)
0923 20:54:03.408312 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.408317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.408503 (+ 186us) server_negotiation.cc:407] Connection header received
0923 20:54:03.408639 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.408643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.408701 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.408783 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.409675 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.410447 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.411664 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.412098 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.412204 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.415880 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:03.415885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:03.415888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:03.415941 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:03.418494 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.419184 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.419187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.419189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.419308 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.419637 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.419640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.419642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.419898 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:03.420103 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:03.420360 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 20:54:03.420617 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260923 20:54:03.425582 6504 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:03.423960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.97.1:33863)
0923 20:54:03.424174 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.424181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.424324 (+ 143us) server_negotiation.cc:407] Connection header received
0923 20:54:03.424463 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.424467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.424543 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.424627 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.425322 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59050: BlockingRecv error: recv got EOF from 127.0.0.1:59050 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260923 20:54:03.425694 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790196790989944-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.427295 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:03.428694 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:54:03.443660 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.430253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.0.97.1:33863)
0923 20:54:03.430436 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.430440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.430778 (+ 338us) server_negotiation.cc:407] Connection header received
0923 20:54:03.430935 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.430939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.431011 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.431095 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.432025 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.432748 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.434216 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.434689 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.434793 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.438423 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:03.438427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:03.438431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:03.438469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:03.441172 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.441810 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.441879 (+ 69us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.441885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.442060 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.442381 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.442384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.442386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.442636 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:03.442862 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:03.443089 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 20:54:03.443255 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:54:03.445366 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6100
I20260923 20:54:03.457790 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.446150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.97.1:33863)
0923 20:54:03.446370 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.446377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.446630 (+ 253us) server_negotiation.cc:407] Connection header received
0923 20:54:03.446751 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.446754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.446811 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.446888 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.447766 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.448434 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.449649 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.450246 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.450349 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.452337 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:03.452342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:03.452345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:03.452379 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:03.455060 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.455815 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.455819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.455822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.455963 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.456328 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.456332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.456333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.456650 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:03.456867 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:03.457189 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 20:54:03.457373 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:54:03.458537 6223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:03.543615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6100
I20260923 20:54:03.560869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6234
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:54:03.577646 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.561905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.97.2:35069)
0923 20:54:03.562474 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.562481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.562508 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:54:03.562570 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.562577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.562791 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.562926 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.563968 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.565070 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.566628 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.567192 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.567309 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.571476 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:03.571491 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:03.571508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:03.571550 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:03.574619 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.575459 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.575467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.575469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.575600 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.576025 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.576030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.576032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.576390 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:03.576716 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:03.577056 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 20:54:03.577326 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":315,"threads_started":1}
I20260923 20:54:03.579028 6357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:03.670159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6234
I20260923 20:54:03.686107 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6368
Sep 23 20:54:03 dist-test-slave-plcg krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:54:03.730229 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:03.687205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.0.97.3:43543)
0923 20:54:03.687921 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:03.687927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:03.687957 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:54:03.688019 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:03.688026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:03.688238 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:03.688464 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:03.689524 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.690704 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.691958 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:03.692550 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:03.692663 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:03.724094 (+ 31431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:03.724110 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:03.724127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:03.724173 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:03.727315 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.728151 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.728160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.728162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.728306 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:03.728664 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:03.728667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:03.728669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:03.729047 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:03.729295 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:03.729528 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 20:54:03.729939 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":402,"threads_started":1}
I20260923 20:54:03.731631 6491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:03.816303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6368
I20260923 20:54:03.832276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6007
I20260923 20:54:03.833647 6069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:03.942150 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6007
2026-09-23T20:54:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4127 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:54:04 dist-test-slave-plcg krb5kdc[6524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:54:04 dist-test-slave-plcg krb5kdc[6524](info): set up 2 sockets
Sep 23 20:54:04 dist-test-slave-plcg krb5kdc[6524](info): commencing operation
krb5kdc: starting...
W20260923 20:54:05.062668 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:54:05 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260923 20:54:05.285688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43651
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:05.619257 6537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:05.619644 6537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:05.619819 6537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:05.627216 6537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:05.627408 6537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:05.627497 6537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:05.627611 6537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:05.627701 6537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:05.637794 6537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:05.639456 6537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:05.640810 6537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:05.653430 6542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:05.653337 6545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:05.653389 6543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:05.654278 6537 server_base.cc:1034] running on GCE node
I20260923 20:54:05.655066 6537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:05.656550 6537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:05.657787 6537 hybrid_clock.cc:648] HybridClock initialized: now 1790196845657718 us; error 69 us; skew 500 ppm
I20260923 20:54:05.658234 6537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:05 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:05.661537 6537 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260923 20:54:05.663381 6537 webserver.cc:460] Webserver started at http://127.0.97.62:39051/ using document root <none> and password file <none>
I20260923 20:54:05.664139 6537 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:05.664306 6537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:05.664633 6537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:05.667804 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "6d08e7f6a2bf48c888ae4860125d0912"
format_stamp: "Formatted at 2026-09-23 20:54:05 on dist-test-slave-plcg"
server_key: "1cb96fdda83bbb2f109add788f432a20"
server_key_iv: "2413a78b14decbeec2287bdf4f6a18a8"
server_key_version: "encryptionkey@0"
I20260923 20:54:05.668635 6537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d08e7f6a2bf48c888ae4860125d0912"
format_stamp: "Formatted at 2026-09-23 20:54:05 on dist-test-slave-plcg"
server_key: "1cb96fdda83bbb2f109add788f432a20"
server_key_iv: "2413a78b14decbeec2287bdf4f6a18a8"
server_key_version: "encryptionkey@0"
I20260923 20:54:05.674530 6537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:54:05.678328 6552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:05.679879 6537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:54:05.680107 6537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "6d08e7f6a2bf48c888ae4860125d0912"
format_stamp: "Formatted at 2026-09-23 20:54:05 on dist-test-slave-plcg"
server_key: "1cb96fdda83bbb2f109add788f432a20"
server_key_iv: "2413a78b14decbeec2287bdf4f6a18a8"
server_key_version: "encryptionkey@0"
I20260923 20:54:05.680351 6537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:05.698614 6537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:05.703524 6537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:05.703878 6537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:05.723948 6537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43651
I20260923 20:54:05.723986 6603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43651 every 8 connection(s)
I20260923 20:54:05.725445 6537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:05.728217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6537
I20260923 20:54:05.728436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:54:05.729240 388 external_mini_cluster.cc:1428] Setting key 369345f7821191053ab0f752a569000a
I20260923 20:54:05.731387 6604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:54:05 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:05.747495 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: Bootstrap starting.
I20260923 20:54:05.751331 6604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:05.752471 6604 log.cc:824] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:05.755626 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:05.732000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.0.97.62:43651)
0923 20:54:05.733073 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:05.733089 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:05.733474 (+ 385us) server_negotiation.cc:407] Connection header received
0923 20:54:05.735386 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:05.735414 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:05.735871 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:05.736273 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:05.738279 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:05.739744 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:05.741591 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:05.742431 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:05.742579 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:05.747608 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:05.747617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:05.747635 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:05.747734 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:05.751004 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:05.751892 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:05.751906 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:05.751914 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:05.752096 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:05.752513 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:05.752519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:05.752522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:05.753347 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:05.753693 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:05.754131 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 20:54:05.754555 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":447,"threads_started":1}
I20260923 20:54:05.756008 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: No bootstrap required, opened a new log
I20260923 20:54:05.763991 6604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } }
I20260923 20:54:05.764457 6604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:05.764576 6604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d08e7f6a2bf48c888ae4860125d0912, State: Initialized, Role: FOLLOWER
I20260923 20:54:05.765234 6604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } }
I20260923 20:54:05.765493 6604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:05.765646 6604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:05.765857 6604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:05.768132 6604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } }
I20260923 20:54:05.768708 6604 leader_election.cc:302] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d08e7f6a2bf48c888ae4860125d0912; no voters:
I20260923 20:54:05.769326 6604 leader_election.cc:288] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:54:05.769467 6609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:05.770274 6609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 1 LEADER]: Becoming Leader. State: Replica: 6d08e7f6a2bf48c888ae4860125d0912, State: Running, Role: LEADER
I20260923 20:54:05.770860 6609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } }
I20260923 20:54:05.771586 6604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:05.773954 6611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d08e7f6a2bf48c888ae4860125d0912. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } }
I20260923 20:54:05.774355 6611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:05.774945 6610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } }
I20260923 20:54:05.775269 6610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:05.776412 6618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:05.781481 6618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:05.791029 6618 catalog_manager.cc:1347] Generated new cluster ID: bebeaef8373045cbba5fa1858e3cfb24
I20260923 20:54:05.791185 6618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:05.812181 6618 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:54:05.813668 6618 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:05.822531 6618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: Generated new TSK 0
I20260923 20:54:05.823503 6618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260923 20:54:05.890125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43651
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:06.265501 6630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:06.265789 6630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:06.265982 6630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:06.273965 6630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:06.274089 6630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:06.274292 6630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:06.285090 6630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:06.286592 6630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:06.287977 6630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:06.302202 6635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:06.302131 6638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:06.302222 6636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:06.302920 6630 server_base.cc:1034] running on GCE node
I20260923 20:54:06.303644 6630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:06.304684 6630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:06.305991 6630 hybrid_clock.cc:648] HybridClock initialized: now 1790196846305936 us; error 39 us; skew 500 ppm
I20260923 20:54:06.306394 6630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:06 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:06.310002 6630 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 20:54:06.311570 6630 webserver.cc:460] Webserver started at http://127.0.97.1:40129/ using document root <none> and password file <none>
I20260923 20:54:06.312342 6630 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:06.312513 6630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:06.312861 6630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:06.316140 6630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "16771be618554d49947f6fc449f5053b"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "182bbc7983c2f79f5f0fd65036af0ded"
server_key_iv: "10823b1b749857395147140dbead303a"
server_key_version: "encryptionkey@0"
I20260923 20:54:06.317000 6630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "16771be618554d49947f6fc449f5053b"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "182bbc7983c2f79f5f0fd65036af0ded"
server_key_iv: "10823b1b749857395147140dbead303a"
server_key_version: "encryptionkey@0"
I20260923 20:54:06.322501 6630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:54:06.326277 6645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:06.327767 6630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:54:06.327986 6630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "16771be618554d49947f6fc449f5053b"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "182bbc7983c2f79f5f0fd65036af0ded"
server_key_iv: "10823b1b749857395147140dbead303a"
server_key_version: "encryptionkey@0"
I20260923 20:54:06.328218 6630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:06.353699 6630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:06.359136 6630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:06.359524 6630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:06.360607 6630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:06.362181 6630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:06.362304 6630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:06.362461 6630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:06.362541 6630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:06.399502 6630 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41167
I20260923 20:54:06.399554 6757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41167 every 8 connection(s)
I20260923 20:54:06.400863 6630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:06.407603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6630
I20260923 20:54:06.407840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:54:06.408491 388 external_mini_cluster.cc:1428] Setting key 32019653a9e8ddb57525fc7a1c8527c7
Sep 23 20:54:06 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:06.424546 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:06.403753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38169 (local address 127.0.97.62:43651)
0923 20:54:06.404302 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:06.404308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:06.405490 (+ 1182us) server_negotiation.cc:407] Connection header received
0923 20:54:06.407574 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:06.407583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:06.407702 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:06.407850 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:06.409794 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.410702 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.412386 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.413168 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.413320 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:06.417651 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:06.417657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:06.417661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:06.417697 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:06.420525 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:06.421687 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:06.421692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:06.421695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:06.421915 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:06.422548 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:06.422554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:06.422557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:06.422850 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:06.422971 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:06.424042 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0923 20:54:06.424262 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":313,"threads_started":1}
I20260923 20:54:06.426244 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:06.404480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.1:38169)
0923 20:54:06.405286 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:06.405338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:06.407215 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:06.408175 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:06.408185 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:06.408842 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:06.409618 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:06.409639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.410882 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.410885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:06.412192 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:06.412206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.413438 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.413444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:06.413667 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:06.414516 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:06.414541 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:06.417337 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:06.420744 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:06.420759 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:06.420782 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:06.421511 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:06.422059 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:06.422066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:06.422073 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:06.422364 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:06.423026 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:06.423038 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:06.423565 (+ 527us) client_negotiation.cc:769] Sending connection context
0923 20:54:06.423904 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 20:54:06.424539 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":323,"threads_started":1}
I20260923 20:54:06.428208 6758 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43651
I20260923 20:54:06.428574 6758 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:06.429375 6758 heartbeater.cc:511] Master 127.0.97.62:43651 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:54:06.432027 6569 ts_manager.cc:194] Registered new tserver with Master: 16771be618554d49947f6fc449f5053b (127.0.97.1:41167)
I20260923 20:54:06.433696 6569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260923 20:54:06.462301 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43651
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:06.814723 6764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:06.815021 6764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:06.815114 6764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:06.822455 6764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:06.822636 6764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:06.822805 6764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:06.833153 6764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:06.834570 6764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:06.835882 6764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:06.849741 6769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:06.849712 6772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:06.849766 6770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:06.850765 6764 server_base.cc:1034] running on GCE node
I20260923 20:54:06.851464 6764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:06.852451 6764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:06.853665 6764 hybrid_clock.cc:648] HybridClock initialized: now 1790196846853600 us; error 63 us; skew 500 ppm
I20260923 20:54:06.854048 6764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:06 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:06.857350 6764 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 20:54:06.859146 6764 webserver.cc:460] Webserver started at http://127.0.97.2:36159/ using document root <none> and password file <none>
I20260923 20:54:06.859933 6764 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:06.860049 6764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:06.860404 6764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:06.863615 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "dece15634329499a8b2456f9ceae3f0d"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "0e2a7a5bef40206039dd14b2fc7e78f2"
server_key_iv: "ed374f9102797d2e45b2a783beae6ac8"
server_key_version: "encryptionkey@0"
I20260923 20:54:06.864491 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dece15634329499a8b2456f9ceae3f0d"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "0e2a7a5bef40206039dd14b2fc7e78f2"
server_key_iv: "ed374f9102797d2e45b2a783beae6ac8"
server_key_version: "encryptionkey@0"
I20260923 20:54:06.870211 6764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:54:06.874404 6779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:06.876195 6764 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:54:06.876407 6764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "dece15634329499a8b2456f9ceae3f0d"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "0e2a7a5bef40206039dd14b2fc7e78f2"
server_key_iv: "ed374f9102797d2e45b2a783beae6ac8"
server_key_version: "encryptionkey@0"
I20260923 20:54:06.876652 6764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:06.908849 6764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:06.913688 6764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:06.914060 6764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:06.915118 6764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:06.916620 6764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:06.916703 6764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:06.916844 6764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:06.916916 6764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:06.951336 6764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38083
I20260923 20:54:06.951409 6891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38083 every 8 connection(s)
I20260923 20:54:06.952699 6764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:06.955461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6764
I20260923 20:54:06.955719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:54:06.956336 388 external_mini_cluster.cc:1428] Setting key 24005071c56a0a4a13f73e98d65452d8
Sep 23 20:54:06 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:06.978114 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:06.955526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42061 (local address 127.0.97.62:43651)
0923 20:54:06.956128 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:06.956135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:06.957479 (+ 1344us) server_negotiation.cc:407] Connection header received
0923 20:54:06.959403 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:06.959407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:06.959481 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:06.959580 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:06.961946 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.962710 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.965409 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.966225 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.966372 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:06.970451 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:06.970460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:06.970468 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:06.970521 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:06.974342 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:06.975238 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:06.975247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:06.975253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:06.975427 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:06.975940 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:06.975946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:06.975949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:06.976357 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:06.976514 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:06.977265 (+ 751us) server_negotiation.cc:299] Negotiation successful
0923 20:54:06.977547 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":315,"threads_started":1}
I20260923 20:54:06.978807 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:06.956153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.2:42061)
0923 20:54:06.957018 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:06.957074 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:06.959050 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:06.959894 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:06.959915 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:06.960772 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:06.961649 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:06.961678 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.962911 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.962919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:06.965210 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:06.965232 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:06.966497 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:06.966503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:06.966723 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:06.967438 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:06.967461 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:06.970131 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:06.974538 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:06.974548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:06.974593 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:06.975092 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:06.975568 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:06.975571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:06.975574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:06.975835 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:06.976508 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:06.976515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:06.976870 (+ 355us) client_negotiation.cc:769] Sending connection context
0923 20:54:06.977113 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 20:54:06.977542 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":381,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:54:06.980635 6892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43651
I20260923 20:54:06.981006 6892 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:06.981894 6892 heartbeater.cc:511] Master 127.0.97.62:43651 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260923 20:54:06.983543 6569 ts_manager.cc:194] Registered new tserver with Master: dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083)
I20260923 20:54:06.984789 6569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 20:54:07.010402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43651
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:07.352619 6898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:07.352931 6898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:07.353133 6898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:07.360702 6898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:07.360831 6898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:07.361037 6898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:07.371052 6898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:07.372462 6898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:07.374037 6898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:07.388579 6904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:07.388631 6903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:07.388664 6906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:07.389349 6898 server_base.cc:1034] running on GCE node
I20260923 20:54:07.390156 6898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:07.391245 6898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:07.392470 6898 hybrid_clock.cc:648] HybridClock initialized: now 1790196847392424 us; error 43 us; skew 500 ppm
I20260923 20:54:07.392822 6898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:07 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:07.396312 6898 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 20:54:07.397989 6898 webserver.cc:460] Webserver started at http://127.0.97.3:45745/ using document root <none> and password file <none>
I20260923 20:54:07.398754 6898 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:07.398924 6898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:07.399255 6898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:07.402472 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "8ee911db158c4ca6bf09fee02e19da8b"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "87881f2dfc34bca62d05f19d4b2cf494"
server_key_iv: "7f8bc03a5771cba2cd3a0806766b77b9"
server_key_version: "encryptionkey@0"
I20260923 20:54:07.403635 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8ee911db158c4ca6bf09fee02e19da8b"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "87881f2dfc34bca62d05f19d4b2cf494"
server_key_iv: "7f8bc03a5771cba2cd3a0806766b77b9"
server_key_version: "encryptionkey@0"
I20260923 20:54:07.409617 6898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:54:07.413388 6913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:07.415234 6898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:07.415431 6898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "8ee911db158c4ca6bf09fee02e19da8b"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "87881f2dfc34bca62d05f19d4b2cf494"
server_key_iv: "7f8bc03a5771cba2cd3a0806766b77b9"
server_key_version: "encryptionkey@0"
I20260923 20:54:07.415655 6898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:07.437222 6758 heartbeater.cc:503] Master 127.0.97.62:43651 was elected leader, sending a full tablet report...
I20260923 20:54:07.445559 6898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:07.450834 6898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:07.451164 6898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:07.452244 6898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:07.453749 6898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:07.453904 6898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:54:07.454083 6898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:07.454151 6898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:07.488888 6898 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34533
I20260923 20:54:07.488931 7025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34533 every 8 connection(s)
I20260923 20:54:07.490401 6898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:07.492511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 6898
I20260923 20:54:07.492717 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:54:07.493225 388 external_mini_cluster.cc:1428] Setting key ada23507d61e968c072fdbb76106debe
Sep 23 20:54:07 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:07.511780 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:07.493344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56067 (local address 127.0.97.62:43651)
0923 20:54:07.493915 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:07.493920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:07.495308 (+ 1388us) server_negotiation.cc:407] Connection header received
0923 20:54:07.497089 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:07.497093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:07.497174 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:07.497259 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:07.499051 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:07.499801 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:07.501527 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:07.502083 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:07.502226 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:07.505723 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:07.505728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:07.505731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:07.505767 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:07.508921 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:07.509672 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:07.509676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:07.509678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:07.509795 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:07.510250 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:07.510253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:07.510255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:07.510521 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:07.510621 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:07.511312 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 20:54:07.511500 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":242,"threads_started":1}
I20260923 20:54:07.512794 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:07.494087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:56067)
0923 20:54:07.494950 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:07.495006 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:07.496842 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:07.497481 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:07.497490 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:07.498208 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:07.498891 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:07.498910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:07.500033 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:07.500038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:07.501352 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:07.501378 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:07.502316 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:07.502321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:07.502533 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:07.503195 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:07.503219 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:07.505477 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:07.509066 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:07.509075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:07.509090 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:07.509552 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:07.509952 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:07.509956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:07.509958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:07.510149 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:07.510638 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:07.510644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:07.510991 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 20:54:07.511223 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 20:54:07.511626 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":373,"threads_started":1}
I20260923 20:54:07.514338 7026 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43651
I20260923 20:54:07.514679 7026 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:07.515427 7026 heartbeater.cc:511] Master 127.0.97.62:43651 requested a full tablet report, sending...
I20260923 20:54:07.517004 6569 ts_manager.cc:194] Registered new tserver with Master: 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533)
I20260923 20:54:07.517899 6569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56067
I20260923 20:54:07.521337 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260923 20:54:07.570356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43389
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43651,127.0.97.61:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:07.916276 7032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:07.916657 7032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:07.916841 7032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:07.924625 7032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:07.924817 7032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:07.924958 7032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:07.925069 7032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:07.925163 7032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:07.935662 7032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43651,127.0.97.61:43389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
W20260923 20:54:07.936132 7032 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:43651,127.0.97.61:43389'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 20:54:07.937250 7032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:07.938656 7032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:07.951130 7038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:07.951306 7040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:07.951175 7037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:07.952826 7032 server_base.cc:1034] running on GCE node
I20260923 20:54:07.953701 7032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:07.955298 7032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:07.956614 7032 hybrid_clock.cc:648] HybridClock initialized: now 1790196847956528 us; error 53 us; skew 500 ppm
I20260923 20:54:07.957018 7032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:07 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:07.960533 7032 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260923 20:54:07.962301 7032 webserver.cc:460] Webserver started at http://127.0.97.61:45443/ using document root <none> and password file <none>
I20260923 20:54:07.963074 7032 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:07.963220 7032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:07.963557 7032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:07.966827 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/instance:
uuid: "e03b196350184da591531013dfe003d0"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "b9f2423b65d7148d4201f2cb5cfe5710"
server_key_iv: "c682e7d680b8a8f829b847ab2cc30f4d"
server_key_version: "encryptionkey@0"
I20260923 20:54:07.967713 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal/instance:
uuid: "e03b196350184da591531013dfe003d0"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "b9f2423b65d7148d4201f2cb5cfe5710"
server_key_iv: "c682e7d680b8a8f829b847ab2cc30f4d"
server_key_version: "encryptionkey@0"
I20260923 20:54:07.973613 7032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:54:07.977878 7047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:07.979391 7032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:07.979609 7032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
uuid: "e03b196350184da591531013dfe003d0"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "b9f2423b65d7148d4201f2cb5cfe5710"
server_key_iv: "c682e7d680b8a8f829b847ab2cc30f4d"
server_key_version: "encryptionkey@0"
I20260923 20:54:07.979856 7032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:07.988255 6892 heartbeater.cc:503] Master 127.0.97.62:43651 was elected leader, sending a full tablet report...
I20260923 20:54:07.998642 7032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:08.003611 7032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:08.003895 7032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:08.024009 7032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43389
I20260923 20:54:08.024034 7098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43389 every 8 connection(s)
I20260923 20:54:08.025491 7032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
I20260923 20:54:08.029452 7099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:08.032666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 7032
I20260923 20:54:08.038275 7099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:08.056644 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.040978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.0.97.62:43651)
0923 20:54:08.042313 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.042318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.042334 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:54:08.042393 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.042396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.042456 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.042602 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.043801 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.044544 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.046203 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.046613 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.046775 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.050652 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.050658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.050661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.050694 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.053561 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.054457 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.054462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.054464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.054587 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.055033 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.055038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.055040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.055349 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.055529 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.056025 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.056249 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1240,"thread_start_us":1134,"threads_started":1}
I20260923 20:54:08.057600 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.040905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45082)
0923 20:54:08.041685 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.041733 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.041921 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.042672 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.042681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.043352 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.043606 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.043615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.044701 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.044706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.046035 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.046048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.046877 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.046884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.047061 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.047755 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.047780 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.050399 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.053744 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.053758 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.053765 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.054289 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.054728 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.054733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.054736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.054890 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.055495 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.055518 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.055707 (+ 189us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.055909 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.056447 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":180,"thread_start_us":354,"threads_started":1}
I20260923 20:54:08.057631 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.035182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.0.97.61:43389)
0923 20:54:08.036055 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.036072 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.036123 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:54:08.037910 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.037942 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.038387 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.038847 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.040373 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.042446 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.043767 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.044461 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.044683 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.049415 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.049425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.049444 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.049481 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.052835 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.053658 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.053675 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.053683 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.053928 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.054339 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.054345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.054348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.054939 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.055261 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.055722 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.056271 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":614,"thread_start_us":365,"threads_started":1}
I20260923 20:54:08.059727 7099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 20:54:08.072630 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.060616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43260)
0923 20:54:08.060869 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.060887 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.061040 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.061329 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.061333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.061706 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.061900 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.061910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.062861 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.062868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.063723 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.063735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.064442 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.064447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.064666 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.065198 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.065214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.067342 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.070164 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.070169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.070173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.070549 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.070931 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.070938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.070943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.071080 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.071627 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.071632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.071782 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.071939 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.072249 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 20:54:08.072660 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.060913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.0.97.61:43389)
0923 20:54:08.061069 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.061076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.061098 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:54:08.061145 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.061149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.061211 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.061316 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.062030 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.062727 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.063875 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.064287 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.064437 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.067567 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.067572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.067576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.067604 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.069995 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.070677 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.070685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.070689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.070815 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.071192 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.071199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.071202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.071480 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.071664 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.072016 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.072369 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":173,"server-negotiator.queue_time_us":60}
I20260923 20:54:08.077880 7099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Bootstrap starting.
I20260923 20:54:08.081269 7099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:08.082491 7099 log.cc:824] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:08.085651 7099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: No bootstrap required, opened a new log
I20260923 20:54:08.092938 7099 raft_consensus.cc:348] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } }
I20260923 20:54:08.093338 7099 raft_consensus.cc:374] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:08.093477 7099 raft_consensus.cc:729] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03b196350184da591531013dfe003d0, State: Initialized, Role: FOLLOWER
I20260923 20:54:08.094216 7099 consensus_queue.cc:260] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } }
I20260923 20:54:08.095134 7106 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } } }
I20260923 20:54:08.095500 7106 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:54:08.096271 7099 sys_catalog.cc:564] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:54:08.105572 7117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:54:08.105755 7117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:54:08.108577 7032 master_runner.cc:428] Detected that this master e03b196350184da591531013dfe003d0 is joining an existing cluster
I20260923 20:54:08.108685 7032 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:43389
I20260923 20:54:08.130781 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.113074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.0.97.61:43389)
0923 20:54:08.113288 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.113294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.113993 (+ 699us) server_negotiation.cc:407] Connection header received
0923 20:54:08.114222 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.114227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.114298 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.114372 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.115968 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.117149 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.119194 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.120034 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.120141 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.122915 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.122922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.122930 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.122970 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.127271 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.128214 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.128222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.128228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.128386 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.128811 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.128818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.128823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.129312 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.129476 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.130000 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.130290 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:54:08.132155 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.113137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43268)
0923 20:54:08.113906 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.113925 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.114100 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.114573 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.114577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.115363 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.115780 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.115794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.117386 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.117393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.119034 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.119049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.120525 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.120531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.120713 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.121512 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.121536 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.122704 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.127436 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.127445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.127451 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.128094 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.128505 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.128511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.128514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.128704 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.129546 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.129554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.129733 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.131224 (+ 1491us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.131569 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":425,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260923 20:54:08.140836 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.114808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45086)
0923 20:54:08.123285 (+ 8477us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.123313 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.123551 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.124590 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.124596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.125668 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.126055 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.126068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.127599 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.127606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.129328 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.129342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.130690 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.130697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.131079 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.131865 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.131889 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.133310 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.137917 (+ 4607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.137923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.137928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.138503 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.138988 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.138996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.139002 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.139170 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.139842 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.139847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.140065 (+ 218us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.140212 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.140468 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8290,"thread_start_us":8067,"threads_started":1}
I20260923 20:54:08.140967 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.114257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.0.97.62:43651)
0923 20:54:08.114460 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.114468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.123420 (+ 8952us) server_negotiation.cc:407] Connection header received
0923 20:54:08.124144 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.124151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.124248 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.124328 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.126260 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.127375 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.129533 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.130252 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.130360 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.133566 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.133573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.133579 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.133620 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.137664 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.138666 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.138673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.138676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.138837 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.139313 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.139318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.139322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.139702 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.139868 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.140377 (+ 509us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.140631 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 20:54:08.168650 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.149435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43276)
0923 20:54:08.150139 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.150182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.150376 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.150928 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.150938 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.151875 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.152207 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.152223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.153745 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.153753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.155639 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.155661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.156795 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.156805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.157101 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.157997 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.158041 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.161245 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.165525 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.165540 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.165546 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.166160 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.166667 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.166670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.166673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.166808 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.167539 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.167546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.167734 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.167929 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.168193 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":286,"threads_started":1}
I20260923 20:54:08.168708 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.149605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.0.97.61:43389)
0923 20:54:08.149795 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.149801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.150307 (+ 506us) server_negotiation.cc:407] Connection header received
0923 20:54:08.150611 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.150617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.150728 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.150845 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.152401 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.153555 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.155844 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.156563 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.156715 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.161535 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.161545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.161553 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.161609 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.165277 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.166335 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.166341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.166345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.166526 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.167032 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.167038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.167042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.167395 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.167558 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.168088 (+ 530us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.168335 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:54:08.170934 6569 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:43389: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e03b196350184da591531013dfe003d0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 43389
}
attrs {
promote: true
}
}
dest_uuid: "6d08e7f6a2bf48c888ae4860125d0912"
cas_config_opid_index: -1
I20260923 20:54:08.172184 6569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260923 20:54:08.174852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34141
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:54:08.185310 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e03b196350184da591531013dfe003d0 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } }
I20260923 20:54:08.187099 7074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:08.189934 7129 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:43389
I20260923 20:54:08.190801 7032 master.cc:537] Master@127.0.97.61:43389 shutting down...
I20260923 20:54:08.191018 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } }
I20260923 20:54:08.191524 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:08.200085 7074 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Refusing update from remote peer 6d08e7f6a2bf48c888ae4860125d0912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 20:54:08.201313 7129 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:54:08.204613 6553 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 20:54:08.207050 7032 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:08.207712 7032 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:08.208173 7032 tablet_replica.cc:330] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: stopping tablet replica
I20260923 20:54:08.218070 7032 master.cc:552] Master@127.0.97.61:43389 shutdown complete.
I20260923 20:54:08.218250 7032 master_runner.cc:305] Clearing existing system tablet
I20260923 20:54:08.219102 7032 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:08.219300 7032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:08.228852 7032 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.005s sys 0.004s
I20260923 20:54:08.231393 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.230901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43284)
0923 20:54:08.231087 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.231206 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:54:08.231813 6553 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 -> Peer e03b196350184da591531013dfe003d0 (127.0.97.61:43389): Couldn't send request to peer e03b196350184da591531013dfe003d0. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:54:08.234225 7136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:08.234650 7032 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:08.234946 7032 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:54:08.235054 7032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
uuid: "e03b196350184da591531013dfe003d0"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "b9f2423b65d7148d4201f2cb5cfe5710"
server_key_iv: "c682e7d680b8a8f829b847ab2cc30f4d"
server_key_version: "encryptionkey@0"
I20260923 20:54:08.240254 7032 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:08.247852 7032 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:54:08.248072 7032 log.cc:1196] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 20:54:08.248539 7032 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Deleting consensus metadata
I20260923 20:54:08.248828 7032 master_runner.cc:315] Copying system tablet from 127.0.97.62:43651
I20260923 20:54:08.254693 7032 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43651
I20260923 20:54:08.273689 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.255682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.0.97.62:43651)
0923 20:54:08.255876 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.255882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.258438 (+ 2556us) server_negotiation.cc:407] Connection header received
0923 20:54:08.258652 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.258658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.258763 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.258894 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.259992 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.261129 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.263061 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.263764 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.263907 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.266494 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.266501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.266506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.266547 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.270387 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.271347 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.271354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.271358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.271534 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.272040 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.272046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.272049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.272418 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.272572 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.273096 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.273335 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:54:08.273763 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.255682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45096)
0923 20:54:08.258268 (+ 2586us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.258291 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.258495 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.258948 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.258954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.259499 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.259786 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.259799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.261300 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.261307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.262868 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.262887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.263973 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.263982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.264186 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.265068 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.265093 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.266221 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.270599 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.270606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.270611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.271163 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.271673 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.271679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.271682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.271865 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.272552 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.272558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.272743 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.272899 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.273259 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2428,"thread_start_us":2298,"threads_started":1}
I20260923 20:54:08.275135 6589 tablet_copy_service.cc:140] P 6d08e7f6a2bf48c888ae4860125d0912: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e03b196350184da591531013dfe003d0 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:45096)
I20260923 20:54:08.275421 6589 tablet_copy_service.cc:161] P 6d08e7f6a2bf48c888ae4860125d0912: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e03b196350184da591531013dfe003d0 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:45096: session id = e03b196350184da591531013dfe003d0-00000000000000000000000000000000
I20260923 20:54:08.278985 6589 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:54:08.281365 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:54:08.287286 7032 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: tablet copy: Starting download of 0 data blocks...
I20260923 20:54:08.287657 7032 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: tablet copy: Starting download of 1 WAL segments...
I20260923 20:54:08.290395 7032 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:54:08.293982 6589 tablet_copy_service.cc:342] P 6d08e7f6a2bf48c888ae4860125d0912: Request end of tablet copy session e03b196350184da591531013dfe003d0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:45096
I20260923 20:54:08.294214 6589 tablet_copy_service.cc:433] P 6d08e7f6a2bf48c888ae4860125d0912: ending tablet copy session e03b196350184da591531013dfe003d0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e03b196350184da591531013dfe003d0
I20260923 20:54:08.304622 7032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:08.307554 7143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:08.317567 7146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:08.318158 7144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:08.320575 7032 server_base.cc:1034] running on GCE node
I20260923 20:54:08.321007 7032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:08.321695 7032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:08.322924 7032 hybrid_clock.cc:648] HybridClock initialized: now 1790196848322860 us; error 37 us; skew 500 ppm
I20260923 20:54:08.323190 7032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:08.325155 7032 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260923 20:54:08.330869 7032 webserver.cc:460] Webserver started at http://127.0.97.61:40107/ using document root <none> and password file <none>
I20260923 20:54:08.331272 7032 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:08.331395 7032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:08.334105 7032 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260923 20:54:08.335979 7153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:08.336452 7032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260923 20:54:08.336622 7032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
uuid: "e03b196350184da591531013dfe003d0"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "b9f2423b65d7148d4201f2cb5cfe5710"
server_key_iv: "c682e7d680b8a8f829b847ab2cc30f4d"
server_key_version: "encryptionkey@0"
I20260923 20:54:08.336792 7032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:08.358558 7032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:08.362128 7032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:08.384991 7032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43389
I20260923 20:54:08.385033 7204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43389 every 8 connection(s)
I20260923 20:54:08.386227 7032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
I20260923 20:54:08.391359 7205 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:54:08.395569 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Bootstrap starting.
I20260923 20:54:08.417443 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:54:08.418037 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: Bootstrap complete.
I20260923 20:54:08.419303 7205 raft_consensus.cc:348] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } }
I20260923 20:54:08.419587 7205 raft_consensus.cc:729] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e03b196350184da591531013dfe003d0, State: Initialized, Role: LEARNER
I20260923 20:54:08.419844 7205 consensus_queue.cc:260] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } }
I20260923 20:54:08.421047 7205 sys_catalog.cc:564] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:08.420828 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } }
I20260923 20:54:08.421304 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:54:08.432788 7219 catalog_manager.cc:1259] Loaded cluster ID: bebeaef8373045cbba5fa1858e3cfb24
I20260923 20:54:08.432909 7219 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: loading cluster ID for follower catalog manager: success
I20260923 20:54:08.436306 7219 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: acquiring CA information for follower catalog manager: success
I20260923 20:54:08.438800 7219 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:54:08.521211 7026 heartbeater.cc:503] Master 127.0.97.62:43651 was elected leader, sending a full tablet report...
W20260923 20:54:08.629000 7131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:08.629379 7131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:08.629557 7131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:08.637647 7131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:08.637830 7131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:08.637961 7131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:08.638052 7131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:08.638123 7131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:08.648387 7131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:08.650131 7131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:08.651484 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:08.665172 7227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:08.665231 7225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:08.665282 7224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:08.667488 7131 server_base.cc:1034] running on GCE node
I20260923 20:54:08.668354 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:08.669968 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:08.671231 7131 hybrid_clock.cc:648] HybridClock initialized: now 1790196848671195 us; error 53 us; skew 500 ppm
I20260923 20:54:08.671607 7131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:08.675017 7131 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260923 20:54:08.676821 7131 webserver.cc:460] Webserver started at http://127.0.97.60:43919/ using document root <none> and password file <none>
I20260923 20:54:08.677600 7131 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:08.678228 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:08.678591 7131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:08.682019 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/instance:
uuid: "3c52ce98709249a5912a4ef7e67c236c"
format_stamp: "Formatted at 2026-09-23 20:54:08 on dist-test-slave-plcg"
server_key: "7a5991b9fd835e5e8086b6b35e58ace0"
server_key_iv: "aa66114775f8672cab790f150e866ddf"
server_key_version: "encryptionkey@0"
I20260923 20:54:08.682907 7131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal/instance:
uuid: "3c52ce98709249a5912a4ef7e67c236c"
format_stamp: "Formatted at 2026-09-23 20:54:08 on dist-test-slave-plcg"
server_key: "7a5991b9fd835e5e8086b6b35e58ace0"
server_key_iv: "aa66114775f8672cab790f150e866ddf"
server_key_version: "encryptionkey@0"
I20260923 20:54:08.688768 7131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:54:08.692762 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:08.694486 7131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:08.694707 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
uuid: "3c52ce98709249a5912a4ef7e67c236c"
format_stamp: "Formatted at 2026-09-23 20:54:08 on dist-test-slave-plcg"
server_key: "7a5991b9fd835e5e8086b6b35e58ace0"
server_key_iv: "aa66114775f8672cab790f150e866ddf"
server_key_version: "encryptionkey@0"
I20260923 20:54:08.694938 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:08.709061 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.702136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.0.97.61:43389)
0923 20:54:08.703120 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.703125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.703145 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:54:08.703204 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.703208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.703275 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.703379 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:08.703923 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.704922 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.707230 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.708104 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.708228 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.708422 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.708681 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.708790 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":739,"threads_started":1}
I20260923 20:54:08.709077 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.701965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43294)
0923 20:54:08.702140 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.702165 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.702269 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.703466 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.703472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.703511 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:08.703710 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.703720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.705079 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.705085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.707010 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.707028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.708281 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.708287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.708477 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.708547 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.708640 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.708788 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 20:54:08.710345 7180 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 20:54:08.711048 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d08e7f6a2bf48c888ae4860125d0912. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } }
I20260923 20:54:08.711344 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:54:08.732831 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:08.738276 7131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:08.738763 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:08.759956 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34141
I20260923 20:54:08.759979 7286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34141 every 8 connection(s)
I20260923 20:54:08.761482 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
I20260923 20:54:08.764979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 7131
I20260923 20:54:08.766016 7287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:08.775895 7287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:08.793227 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.767562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.0.97.60:34141)
0923 20:54:08.768931 (+ 1369us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.768943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.768995 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:54:08.771451 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.771480 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.771960 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.772445 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.774065 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.775637 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.777103 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.777800 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.778076 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.783990 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.784001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.784017 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.784093 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.788361 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.789351 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.789358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.789365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.789595 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.790152 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.790158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.790161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.790769 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.790999 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.791584 (+ 585us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.791994 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":409,"threads_started":1}
I20260923 20:54:08.797977 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.778817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.0.97.62:43651)
0923 20:54:08.779364 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.779371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.779949 (+ 578us) server_negotiation.cc:407] Connection header received
0923 20:54:08.780164 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.780168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.780238 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.780362 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.782504 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.783328 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.785500 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.786020 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.786164 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.790516 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.790522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.790525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.790560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.794686 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.795738 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.795743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.795745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.795900 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.796438 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.796445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.796449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.796814 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.796940 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.797435 (+ 495us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.797637 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":248,"threads_started":1}
I20260923 20:54:08.798004 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.778764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45104)
0923 20:54:08.779770 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.779822 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.780037 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.780461 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.780470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.781432 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.782307 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.782326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.783495 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.783501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.785330 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.785348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.786259 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.786272 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.786513 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.787119 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.787146 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.790301 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.794874 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.794888 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.794899 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.795577 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.796034 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.796040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.796044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.796270 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.796948 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.796954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.797119 (+ 165us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.797311 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.797598 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":534,"threads_started":1}
I20260923 20:54:08.799623 7287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 20:54:08.813043 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.800451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43304)
0923 20:54:08.800694 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.800713 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.800819 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.801158 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.801162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.801501 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.802153 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.802165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.803168 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.803172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.804119 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.804132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.804881 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.804886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.805007 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.805599 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.805616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.807761 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.810840 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.810844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.810847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.811257 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.811647 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.811651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.811653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.811780 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.812304 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.812308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.812451 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.812576 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.812810 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:54:08.813086 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.800482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.97.61:43389)
0923 20:54:08.800705 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.800711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.800760 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:54:08.800927 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.800930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.800991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.801078 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.802313 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.803017 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.804278 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.804713 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.804814 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.807990 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.807996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.807999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.808036 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.810676 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.811410 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.811414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.811416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.811544 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.811876 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.811879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.811881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.812142 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.812256 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.812666 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.812858 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:54:08.814224 7287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 20:54:08.827803 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.814896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.0.1:32890)
0923 20:54:08.815128 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.815148 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.815238 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.815516 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.815519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.815845 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.816004 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.816012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.816939 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.816951 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.817790 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.817801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.818586 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.818591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.818745 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.819422 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.819471 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.821567 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.825066 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.825070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.825073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.825430 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.825896 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.825903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.825908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.826089 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.826704 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.826709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.826880 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.827033 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.827348 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 20:54:08.827836 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.815122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.0.97.60:34141)
0923 20:54:08.815271 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.815275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.815291 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:54:08.815330 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.815333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.815399 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.815477 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.816125 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.816806 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.817991 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.818410 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.818509 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.821759 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.821764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.821767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.821798 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.824910 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.825549 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.825554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.825588 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.825719 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.826207 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.826212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.826216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.826575 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.826752 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.827157 (+ 405us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.827531 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":53}
I20260923 20:54:08.833061 7287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Bootstrap starting.
I20260923 20:54:08.836354 7287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:08.837451 7287 log.cc:824] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:08.840332 7287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: No bootstrap required, opened a new log
I20260923 20:54:08.848333 7287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } }
I20260923 20:54:08.848675 7287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:08.848759 7287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c52ce98709249a5912a4ef7e67c236c, State: Initialized, Role: FOLLOWER
I20260923 20:54:08.849357 7287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } }
I20260923 20:54:08.850400 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } } }
I20260923 20:54:08.850749 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:54:08.851460 7287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:54:08.861083 7305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:54:08.861250 7305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:54:08.866240 7131 master_runner.cc:428] Detected that this master 3c52ce98709249a5912a4ef7e67c236c is joining an existing cluster
I20260923 20:54:08.866371 7131 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:34141
I20260923 20:54:08.892309 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.871334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.0.97.62:43651)
0923 20:54:08.871572 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.871580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.872733 (+ 1153us) server_negotiation.cc:407] Connection header received
0923 20:54:08.873077 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.873084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.873203 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.873291 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.875320 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.876864 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.880066 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.881077 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.881222 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.885111 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.885120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.885125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.885169 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.888885 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.889803 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.889809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.889833 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.890024 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.890528 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.890533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.890537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.890889 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.891028 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.891632 (+ 604us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.891902 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 20:54:08.894188 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.871783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45106)
0923 20:54:08.872618 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.872642 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.872894 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.873583 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.873591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.874575 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.875095 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.875109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.877233 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.877242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.879781 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.879806 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.881671 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.881682 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.882003 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.883119 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.883156 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.884796 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.889064 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.889069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.889073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.889642 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.890196 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.890201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.890205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.890396 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.891103 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.891108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.891333 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.892602 (+ 1269us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.893558 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":513,"threads_started":1}
I20260923 20:54:08.901226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6630
I20260923 20:54:08.903663 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.873098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.0.1:32896)
0923 20:54:08.885205 (+ 12107us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.885226 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.885433 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.885913 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.885917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.886746 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.887271 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.887284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.888787 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.888792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.890515 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.890530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.891856 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.891863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.892047 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.893592 (+ 1545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.893614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.894910 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.899863 (+ 4953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.899869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.899875 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.900445 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.900873 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.900878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.900882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.901042 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.902028 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.902034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.902237 (+ 203us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.902788 (+ 551us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.903099 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11927,"mutex_wait_us":33,"thread_start_us":11751,"threads_started":1}
I20260923 20:54:08.904769 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.872947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.0.97.60:34141)
0923 20:54:08.873213 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.873222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.885310 (+ 12088us) server_negotiation.cc:407] Connection header received
0923 20:54:08.885559 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.885564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.885631 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.885694 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.887426 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.888555 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.890674 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.891434 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.891534 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.895123 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.895130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.895136 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.895176 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.899699 (+ 4523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.900583 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.900592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.900596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.900744 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.901215 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.901221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.901225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.901886 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.902063 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.904171 (+ 2108us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.904404 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:54:08.909093 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.871868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.97.61:43389)
0923 20:54:08.872376 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.872384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.884306 (+ 11922us) server_negotiation.cc:407] Connection header received
0923 20:54:08.885233 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.885239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.885329 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.885458 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.887455 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.888582 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.890961 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.891656 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.891747 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.895086 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.895092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.895097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.895139 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.903858 (+ 8719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.906647 (+ 2789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.906654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.906658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.906848 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.907308 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.907313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.907317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.907677 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.907811 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.908413 (+ 602us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.908732 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260923 20:54:08.910636 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.871933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43312)
0923 20:54:08.884162 (+ 12229us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.884186 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.885095 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.885718 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.885723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.886655 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.887221 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.887251 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.888797 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.888804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.890778 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.890794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.892049 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.892056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.892293 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.892866 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.892894 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.894858 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.905958 (+ 11100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.905964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.905969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.906477 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.906994 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.906999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.907003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.907163 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.907886 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.907893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.908100 (+ 207us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.909712 (+ 1612us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.910132 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12056,"mutex_wait_us":29,"thread_start_us":12379,"threads_started":1}
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 20:54:08.918546 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.900451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.0.1:32908)
0923 20:54:08.900731 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:08.900752 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:08.900872 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:08.902125 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:08.902131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:08.903073 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:08.903276 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.903289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.905310 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.905316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.906722 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:08.906741 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.907829 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.907837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:08.908045 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.908735 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:08.908758 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:08.911566 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:08.915975 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.915981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.915985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.916369 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.916791 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:08.916796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:08.916800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:08.916956 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:08.917506 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:08.917511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:08.917669 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 20:54:08.917818 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 20:54:08.918154 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260923 20:54:08.918900 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.900501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.97.60:34141)
0923 20:54:08.901681 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.901685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.901703 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:54:08.901773 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.901776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.901911 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.903434 (+ 1523us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.903795 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.905117 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.906913 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.907643 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.911720 (+ 4077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.911923 (+ 203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.911935 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.911942 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.911966 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.915789 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.916517 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.916523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.916527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.916659 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.917078 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.917084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.917087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.917369 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.917547 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.917938 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.918153 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":3013,"threads_started":1}
I20260923 20:54:08.921177 6569 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:34141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 34141
}
attrs {
promote: true
}
}
dest_uuid: "6d08e7f6a2bf48c888ae4860125d0912"
cas_config_opid_index: 5
I20260923 20:54:08.922324 6569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } }
I20260923 20:54:08.924456 7132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3c52ce98709249a5912a4ef7e67c236c (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.925076 7180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 LEARNER]: Refusing update from remote peer 6d08e7f6a2bf48c888ae4860125d0912: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 20:54:08.925259 7262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:08.928280 7132 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:34141
Sep 23 20:54:08 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260923 20:54:08.944299 7316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 20:54:08.944618 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.944839 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:08.948658 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:08.903309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.0.97.1:41167)
0923 20:54:08.913300 (+ 9991us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:08.913313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:08.913350 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:54:08.913426 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:08.913436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:08.913748 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:08.913987 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:08.916183 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.917857 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.920191 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:08.921185 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:08.921324 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:08.935156 (+ 13832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:08.935179 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:08.935207 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:08.935286 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:08.940868 (+ 5582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.942265 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.942279 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.942285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.942521 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:08.943105 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:08.943113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:08.943119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:08.943703 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:08.947385 (+ 3682us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:08.947629 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:54:08.948042 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9751,"thread_start_us":9559,"threads_started":1}
I20260923 20:54:08.950582 6753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:08.958504 7262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Refusing update from remote peer 6d08e7f6a2bf48c888ae4860125d0912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 20:54:08.958586 7131 master.cc:537] Master@127.0.97.60:34141 shutting down...
I20260923 20:54:08.958925 7180 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3c52ce98709249a5912a4ef7e67c236c (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.960544 7132 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:08.963192 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: true } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.963471 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:54:08.964089 7132 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: attempting to promote NON_VOTER e03b196350184da591531013dfe003d0 to VOTER
I20260923 20:54:08.964977 7132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } }
I20260923 20:54:08.970662 7180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 LEARNER]: Refusing update from remote peer 6d08e7f6a2bf48c888ae4860125d0912: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 20:54:08.971401 7317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 20:54:08.973412 7131 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:08.974074 7131 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:08.974325 7131 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: stopping tablet replica
I20260923 20:54:08.975746 7132 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e03b196350184da591531013dfe003d0 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.976557 7179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e03b196350184da591531013dfe003d0 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.980304 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.980608 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:08.981071 7132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.981297 7132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:08.981359 7208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:08.981647 7208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 20:54:08.987265 6555 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 -> Peer 3c52ce98709249a5912a4ef7e67c236c (127.0.97.60:34141): Couldn't send request to peer 3c52ce98709249a5912a4ef7e67c236c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 20:54:08.988962 7131 master.cc:552] Master@127.0.97.60:34141 shutdown complete.
I20260923 20:54:08.989104 7131 master_runner.cc:305] Clearing existing system tablet
I20260923 20:54:08.989566 7131 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:08.989707 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:08.994576 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:54:08.996417 7324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:08.996848 7131 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:08.997190 7131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 20:54:08.997323 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
uuid: "3c52ce98709249a5912a4ef7e67c236c"
format_stamp: "Formatted at 2026-09-23 20:54:08 on dist-test-slave-plcg"
server_key: "7a5991b9fd835e5e8086b6b35e58ace0"
server_key_iv: "aa66114775f8672cab790f150e866ddf"
server_key_version: "encryptionkey@0"
I20260923 20:54:09.002595 7131 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:09.009804 7131 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:54:09.010102 7131 log.cc:1196] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 20:54:09.010833 7131 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Deleting consensus metadata
I20260923 20:54:09.011132 7131 master_runner.cc:315] Copying system tablet from 127.0.97.62:43651
I20260923 20:54:09.013936 7131 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:43651
I20260923 20:54:09.030757 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.015105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.0.97.62:43651)
0923 20:54:09.015313 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:09.015322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:09.015418 (+ 96us) server_negotiation.cc:407] Connection header received
0923 20:54:09.015688 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:09.015694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:09.015783 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:09.015854 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:09.017064 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.018258 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.020171 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.020885 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.021042 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.023483 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:09.023490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:09.023497 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:09.023539 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:09.027368 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.028484 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.028491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.028495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.028664 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.029129 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.029135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.029138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.029488 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:09.029631 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:09.030128 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 20:54:09.030371 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 20:54:09.030979 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.014780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45114)
0923 20:54:09.015307 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:09.015333 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:09.015542 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:09.016069 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:09.016076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:09.016607 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:09.016866 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.016879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.018450 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.018456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.020003 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.020021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.021107 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.021114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.021288 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.022085 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:09.022124 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:09.023216 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:09.027740 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.027746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.027751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.028308 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.028807 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.028812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.028817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.028990 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.029619 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:09.029625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:09.029811 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 20:54:09.030010 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 20:54:09.030345 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":408,"threads_started":1}
I20260923 20:54:09.031822 6589 tablet_copy_service.cc:140] P 6d08e7f6a2bf48c888ae4860125d0912: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c52ce98709249a5912a4ef7e67c236c ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:45114)
I20260923 20:54:09.031998 6589 tablet_copy_service.cc:161] P 6d08e7f6a2bf48c888ae4860125d0912: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c52ce98709249a5912a4ef7e67c236c at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:45114: session id = 3c52ce98709249a5912a4ef7e67c236c-00000000000000000000000000000000
I20260923 20:54:09.035799 6589 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:54:09.037895 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:54:09.045439 7131 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: tablet copy: Starting download of 0 data blocks...
I20260923 20:54:09.045805 7131 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: tablet copy: Starting download of 1 WAL segments...
I20260923 20:54:09.048594 7131 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:54:09.053579 6589 tablet_copy_service.cc:342] P 6d08e7f6a2bf48c888ae4860125d0912: Request end of tablet copy session 3c52ce98709249a5912a4ef7e67c236c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:45114
I20260923 20:54:09.053748 6589 tablet_copy_service.cc:433] P 6d08e7f6a2bf48c888ae4860125d0912: ending tablet copy session 3c52ce98709249a5912a4ef7e67c236c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c52ce98709249a5912a4ef7e67c236c
I20260923 20:54:09.064847 7131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:09.068082 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:09.068903 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:09.073549 7131 server_base.cc:1034] running on GCE node
W20260923 20:54:09.073619 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:09.074271 7131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:09.079041 7131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:09.079420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6630
I20260923 20:54:09.085932 7131 hybrid_clock.cc:648] HybridClock initialized: now 1790196849085813 us; error 98 us; skew 500 ppm
I20260923 20:54:09.086439 7131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:09 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:09.089057 7131 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260923 20:54:09.094509 7131 webserver.cc:460] Webserver started at http://127.0.97.60:37413/ using document root <none> and password file <none>
I20260923 20:54:09.094946 7131 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:09.095124 7131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:09.098810 7131 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 20:54:09.101348 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:09.101971 7131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 20:54:09.102108 7131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
uuid: "3c52ce98709249a5912a4ef7e67c236c"
format_stamp: "Formatted at 2026-09-23 20:54:08 on dist-test-slave-plcg"
server_key: "7a5991b9fd835e5e8086b6b35e58ace0"
server_key_iv: "aa66114775f8672cab790f150e866ddf"
server_key_version: "encryptionkey@0"
I20260923 20:54:09.102314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41167
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43651
--webserver_port=40129
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:54:09.102375 7131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:09.149895 7131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:09.154552 7131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:09.175972 7131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34141
I20260923 20:54:09.176007 7393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34141 every 8 connection(s)
I20260923 20:54:09.177357 7131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
I20260923 20:54:09.182964 7394 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:54:09.187915 7394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Bootstrap starting.
I20260923 20:54:09.209378 7394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:54:09.210141 7394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Bootstrap complete.
I20260923 20:54:09.211416 7394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } }
I20260923 20:54:09.211737 7394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c52ce98709249a5912a4ef7e67c236c, State: Initialized, Role: LEARNER
I20260923 20:54:09.212149 7394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } }
I20260923 20:54:09.213490 7394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:09.213124 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:09.213721 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: This master's current role is: LEARNER
I20260923 20:54:09.225328 7408 catalog_manager.cc:1259] Loaded cluster ID: bebeaef8373045cbba5fa1858e3cfb24
I20260923 20:54:09.225464 7408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: loading cluster ID for follower catalog manager: success
I20260923 20:54:09.229319 7408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: acquiring CA information for follower catalog manager: success
I20260923 20:54:09.231999 7408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:54:09.393713 7409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.387817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.0.97.60:34141)
0923 20:54:09.388398 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:09.388402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:09.388421 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:54:09.388480 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:09.388483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:09.388548 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:09.388705 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:09.389100 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.390187 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.391769 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.392605 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.392771 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.393024 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:09.393225 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 20:54:09.393361 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":341,"threads_started":1}
I20260923 20:54:09.393702 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.387659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.0.1:32916)
0923 20:54:09.387810 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:09.387832 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:09.387941 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:09.388726 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:09.388730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:09.388758 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:09.388940 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.388950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.390346 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.390352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.391578 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.391591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.392798 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.392804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.393013 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.393051 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:54:09.393206 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 20:54:09.393361 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 20:54:09.394804 7369 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 20:54:09.395447 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d08e7f6a2bf48c888ae4860125d0912. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:09.395759 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: This master's current role is: LEARNER
W20260923 20:54:09.506459 7346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:09.506769 7346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:09.506865 7346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:09.514876 7346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:09.515057 7346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:09.515184 7346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:09.526715 7346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40129
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:09.528216 7346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:09.529557 7346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:09.545472 7415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:09.545533 7417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:09.545503 7414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:09.547178 7346 server_base.cc:1034] running on GCE node
I20260923 20:54:09.547897 7346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:09.548949 7346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:09.550194 7346 hybrid_clock.cc:648] HybridClock initialized: now 1790196849550120 us; error 64 us; skew 500 ppm
I20260923 20:54:09.550506 7346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:09 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:09.554400 7346 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 20:54:09.556066 7346 webserver.cc:460] Webserver started at http://127.0.97.1:40129/ using document root <none> and password file <none>
I20260923 20:54:09.556860 7346 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:09.557030 7346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:09.563123 7346 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:54:09.566753 7424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:09.568764 7346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:54:09.569012 7346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "16771be618554d49947f6fc449f5053b"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "182bbc7983c2f79f5f0fd65036af0ded"
server_key_iv: "10823b1b749857395147140dbead303a"
server_key_version: "encryptionkey@0"
I20260923 20:54:09.570269 7346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:09.592334 7346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:09.597208 7346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:09.597558 7346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:09.598687 7346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:09.600171 7346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:09.600324 7346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:09.600476 7346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:09.600571 7346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:09.641978 7346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41167
I20260923 20:54:09.642026 7536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41167 every 8 connection(s)
I20260923 20:54:09.644045 7346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:09.652755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 7346
I20260923 20:54:09.653165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6764
Sep 23 20:54:09 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 23 20:54:09 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 20:54:09.684264 7409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.648704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35183 (local address 127.0.97.60:34141)
0923 20:54:09.648960 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:09.648970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:09.656748 (+ 7778us) server_negotiation.cc:407] Connection header received
0923 20:54:09.656844 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:09.656853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:09.656952 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:09.657045 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:09.660521 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.661919 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.666910 (+ 4991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.667844 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.667982 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.674334 (+ 6352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:09.674343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:09.674348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:09.674395 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:09.678387 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.680886 (+ 2499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.680892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.680895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.681086 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.681704 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.681710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.681714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.682165 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:09.682316 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:09.683614 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0923 20:54:09.683883 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Sep 23 20:54:09 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:09.686878 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.649510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.97.1:35183)
0923 20:54:09.652730 (+ 3220us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:09.652784 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:09.656197 (+ 3413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:09.657568 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:09.657586 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:09.658659 (+ 1073us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:09.660230 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.660260 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.662331 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.662361 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.665748 (+ 3387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.665777 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.668489 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.668499 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.668945 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.669977 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:09.670031 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:09.673956 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:09.678576 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.678586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.678608 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.680727 (+ 2119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.681258 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.681265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.681271 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.681550 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.682426 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:09.682434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:09.683122 (+ 688us) client_negotiation.cc:769] Sending connection context
0923 20:54:09.684520 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0923 20:54:09.685249 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2848,"thread_start_us":2560,"threads_started":1}
I20260923 20:54:09.688263 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.648512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52569 (local address 127.0.97.61:43389)
0923 20:54:09.650504 (+ 1992us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:09.650529 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:09.666663 (+ 16134us) server_negotiation.cc:407] Connection header received
0923 20:54:09.666855 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:09.666864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:09.666992 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:09.667149 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:09.668847 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.670291 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.673180 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.674125 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.674255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.679647 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:09.679653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:09.679658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:09.679697 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:09.684559 (+ 4862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.685771 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.685780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.685785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.685976 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.686433 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.686439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.686444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.686838 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:09.686967 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:09.687542 (+ 575us) server_negotiation.cc:299] Negotiation successful
0923 20:54:09.687776 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1820,"thread_start_us":1697,"threads_started":1}
I20260923 20:54:09.689325 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.650866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.97.1:52569)
0923 20:54:09.666490 (+ 15624us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:09.666516 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:09.666698 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:09.667460 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:09.667467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:09.668233 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:09.668622 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.668637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.670553 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.670561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.672729 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.672905 (+ 176us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.674723 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.674731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.675041 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.675820 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:09.675849 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:09.679372 (+ 3523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:09.684778 (+ 5406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.684783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.684787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.685548 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.686150 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.686155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.686159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.686302 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.687077 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:09.687084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:09.687254 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 20:54:09.688494 (+ 1240us) client_negotiation.cc:240] Negotiation successful
0923 20:54:09.688844 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15422,"thread_start_us":15167,"threads_started":1}
I20260923 20:54:09.689795 7537 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34141
I20260923 20:54:09.690124 7537 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:09.690341 7539 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43389
I20260923 20:54:09.690554 7539 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:09.691332 7539 heartbeater.cc:511] Master 127.0.97.61:43389 requested a full tablet report, sending...
I20260923 20:54:09.691362 7537 heartbeater.cc:511] Master 127.0.97.60:34141 requested a full tablet report, sending...
I20260923 20:54:09.694101 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.656294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.1:55993)
0923 20:54:09.671758 (+ 15464us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:09.671783 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:09.672707 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:09.673927 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:09.673934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:09.674545 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:09.674924 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.674938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.677739 (+ 2801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.677744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.679937 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:09.679952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.681254 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.681261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:09.681462 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.682317 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:09.682343 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:09.685563 (+ 3220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:09.690998 (+ 5435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.691003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.691008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.691567 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.692050 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:09.692056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:09.692062 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:09.692246 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:09.692993 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:09.692997 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:09.693163 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 20:54:09.693335 (+ 172us) client_negotiation.cc:240] Negotiation successful
0923 20:54:09.693656 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15288,"thread_start_us":15578,"threads_started":1}
I20260923 20:54:09.694139 7170 ts_manager.cc:194] Registered new tserver with Master: 16771be618554d49947f6fc449f5053b (127.0.97.1:41167)
Sep 23 20:54:09 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260923 20:54:09.694237 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.653642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55993 (local address 127.0.97.62:43651)
0923 20:54:09.673533 (+ 19891us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:09.673540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:09.673560 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:54:09.673623 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:09.673628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:09.673718 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:09.675757 (+ 2039us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:09.676038 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.677544 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.680112 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.680761 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.680917 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.685806 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:09.685836 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:09.685841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:09.685887 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:09.690803 (+ 4916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.691743 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.691749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.691752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.691900 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.692375 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.692380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.692383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.692770 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:09.692903 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:09.693552 (+ 649us) server_negotiation.cc:299] Negotiation successful
0923 20:54:09.693809 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19748,"thread_start_us":21956,"threads_started":1}
I20260923 20:54:09.695370 7538 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43651
I20260923 20:54:09.695525 7538 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:09.695936 7538 heartbeater.cc:511] Master 127.0.97.62:43651 requested a full tablet report, sending...
I20260923 20:54:09.696105 7359 ts_manager.cc:194] Registered new tserver with Master: 16771be618554d49947f6fc449f5053b (127.0.97.1:41167)
I20260923 20:54:09.697120 6569 ts_manager.cc:194] Re-registered known tserver with Master: 16771be618554d49947f6fc449f5053b (127.0.97.1:41167)
I20260923 20:54:09.698310 6569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55993
I20260923 20:54:09.704789 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:09.654396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.0.97.2:38083)
0923 20:54:09.679873 (+ 25477us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:09.679883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:09.679917 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:54:09.679985 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:09.679997 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:09.680310 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:09.682982 (+ 2672us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:09.683247 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.684884 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.687025 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:09.688811 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:09.688945 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:09.697877 (+ 8932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:09.697897 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:09.697918 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:09.697965 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:09.701633 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.702496 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.702503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.702505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.702721 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:09.703146 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:09.703149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:09.703151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:09.703489 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:09.703797 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:09.704144 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 20:54:09.704350 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25289,"thread_start_us":25106,"threads_started":1}
I20260923 20:54:09.706183 6887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:09.792485 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6764
I20260923 20:54:09.809616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38083
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43651
--webserver_port=36159
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:54:09.968990 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: true } } }
I20260923 20:54:09.969331 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:09.971237 7550 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: attempting to promote NON_VOTER 3c52ce98709249a5912a4ef7e67c236c to VOTER
I20260923 20:54:09.971930 7550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } }
I20260923 20:54:09.974026 7369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 LEARNER]: Refusing update from remote peer 6d08e7f6a2bf48c888ae4860125d0912: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:54:09.974092 7179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Refusing update from remote peer 6d08e7f6a2bf48c888ae4860125d0912: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:54:09.974828 7322 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:54:09.975302 7551 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:54:09.978449 7322 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c52ce98709249a5912a4ef7e67c236c (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.979218 7179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c52ce98709249a5912a4ef7e67c236c (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.985423 7550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.986461 7550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:09.986095 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.986799 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:09.987480 7554 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.988083 7554 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:54:09.992297 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.992698 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:09.993188 7369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c52ce98709249a5912a4ef7e67c236c (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.996753 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d08e7f6a2bf48c888ae4860125d0912" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:09.997336 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: This master's current role is: FOLLOWER
W20260923 20:54:10.168828 7549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:10.169221 7549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:10.169384 7549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:10.177480 7549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:10.177664 7549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:10.177913 7549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:10.188999 7549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36159
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:10.190562 7549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:10.191965 7549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:10.207144 7560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:10.207126 7561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:10.207360 7563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:10.208138 7549 server_base.cc:1034] running on GCE node
I20260923 20:54:10.208840 7549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:10.209920 7549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:10.211211 7549 hybrid_clock.cc:648] HybridClock initialized: now 1790196850211155 us; error 64 us; skew 500 ppm
I20260923 20:54:10.211598 7549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:10.215102 7549 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 20:54:10.216697 7549 webserver.cc:460] Webserver started at http://127.0.97.2:36159/ using document root <none> and password file <none>
I20260923 20:54:10.217465 7549 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:10.217661 7549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:10.225029 7549 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 20:54:10.228579 7570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:10.230283 7549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:54:10.230504 7549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "dece15634329499a8b2456f9ceae3f0d"
format_stamp: "Formatted at 2026-09-23 20:54:06 on dist-test-slave-plcg"
server_key: "0e2a7a5bef40206039dd14b2fc7e78f2"
server_key_iv: "ed374f9102797d2e45b2a783beae6ac8"
server_key_version: "encryptionkey@0"
I20260923 20:54:10.231580 7549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:10.268620 7549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:10.273406 7549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:10.273770 7549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:10.275207 7549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:10.276897 7549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:10.276994 7549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:10.277127 7549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:10.277227 7549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:10.312253 7549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38083
I20260923 20:54:10.312309 7682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38083 every 8 connection(s)
I20260923 20:54:10.313822 7549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:10.315915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 7549
I20260923 20:54:10.316202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6898
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260923 20:54:10.345739 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.316914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41775 (local address 127.0.97.62:43651)
0923 20:54:10.317480 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.317486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.324978 (+ 7492us) server_negotiation.cc:407] Connection header received
0923 20:54:10.325052 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.325057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.325156 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.325269 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.326512 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.327721 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.330153 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.330848 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.331008 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.336688 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:10.336698 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:10.336703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:10.336775 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:10.341273 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.342672 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.342679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.342682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.342855 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.343470 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.343474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.343477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.343824 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:10.343956 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:10.345111 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0923 20:54:10.345354 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260923 20:54:10.346000 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.318355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45433 (local address 127.0.97.61:43389)
0923 20:54:10.322186 (+ 3831us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.322195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.322220 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:54:10.322300 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.322308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.322413 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.322536 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.324959 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.326783 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.329858 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.330507 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.330654 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.337876 (+ 7222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:10.337887 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:10.337894 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:10.337956 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:10.342095 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.343061 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.343067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.343072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.343232 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.343673 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.343678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.343680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.344045 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:10.345227 (+ 1182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:10.345354 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 20:54:10.345607 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3667,"thread_start_us":3531,"threads_started":1}
I20260923 20:54:10.347258 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.318898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.2:41775)
0923 20:54:10.322708 (+ 3810us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:10.322726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:10.322844 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:10.325294 (+ 2450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:10.325298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:10.326056 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:10.326359 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.326370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.328137 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.328147 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.330007 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.330024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.331078 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.331086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.331276 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.332076 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:10.332125 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:10.336339 (+ 4214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:10.341500 (+ 5161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.341515 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.341539 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.342512 (+ 973us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.343008 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.343012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.343016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.343346 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.344045 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:10.344054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:10.344629 (+ 575us) client_negotiation.cc:769] Sending connection context
0923 20:54:10.346051 (+ 1422us) client_negotiation.cc:240] Negotiation successful
0923 20:54:10.346476 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3678,"mutex_wait_us":175,"spinlock_wait_cycles":62464,"thread_start_us":3484,"threads_started":1}
I20260923 20:54:10.347353 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.317701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.97.2:45433)
0923 20:54:10.318681 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:10.318733 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:10.321937 (+ 3204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:10.322878 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:10.322893 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:10.323699 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:10.324743 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.324761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.327027 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.327033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.329625 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.329641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.330794 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.330802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.331144 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.332180 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:10.332210 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:10.337607 (+ 5397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:10.342290 (+ 4683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.342297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.342307 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.342874 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.343370 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.343375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.343378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.343536 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.344208 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:10.344213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:10.344629 (+ 416us) client_negotiation.cc:769] Sending connection context
0923 20:54:10.344968 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 20:54:10.346778 (+ 1810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":204,"spinlock_wait_cycles":78464,"thread_start_us":376,"threads_started":1}
I20260923 20:54:10.347990 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.323014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37893 (local address 127.0.97.60:34141)
0923 20:54:10.323529 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.323535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.326178 (+ 2643us) server_negotiation.cc:407] Connection header received
0923 20:54:10.326252 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.326296 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.326383 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.327066 (+ 683us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.328557 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.329717 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.331566 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.332298 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.332472 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.336816 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:10.336826 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:10.336833 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:10.336880 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:10.341455 (+ 4575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.342672 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.342679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.342682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.342878 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.343452 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.343456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.343459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.343818 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:10.343974 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:10.347351 (+ 3377us) server_negotiation.cc:299] Negotiation successful
0923 20:54:10.347601 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":494,"threads_started":1}
I20260923 20:54:10.349108 7685 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43389
I20260923 20:54:10.349232 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.323379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.97.2:37893)
0923 20:54:10.325911 (+ 2532us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:10.325932 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:10.326052 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:10.327252 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:10.327261 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:10.327874 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:10.328346 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.328362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.329945 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.329950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.331356 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.331374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.332683 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.332690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.332901 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.333639 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:10.333665 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:10.336529 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:10.341603 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.341610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.341615 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.342494 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.343008 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.343013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.343016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.343299 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.344045 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:10.344054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:10.344629 (+ 575us) client_negotiation.cc:769] Sending connection context
0923 20:54:10.344969 (+ 340us) client_negotiation.cc:240] Negotiation successful
0923 20:54:10.345514 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"mutex_wait_us":102,"spinlock_wait_cycles":14208,"thread_start_us":2168,"threads_started":1}
I20260923 20:54:10.349478 7685 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:10.349960 7684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43651
I20260923 20:54:10.349997 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.318894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.0.97.3:34533)
0923 20:54:10.321021 (+ 2127us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.321033 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.321690 (+ 657us) server_negotiation.cc:407] Connection header received
0923 20:54:10.322990 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.323000 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.323281 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.323435 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.326277 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.327910 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.329851 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.332018 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.332159 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.340125 (+ 7966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:10.340149 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:10.340171 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:10.340233 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:10.344302 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.345252 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.345261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.345264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.345424 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.346427 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.346432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.346434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.347080 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:10.347931 (+ 851us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:10.349283 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0923 20:54:10.349592 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1912,"thread_start_us":1734,"threads_started":1}
I20260923 20:54:10.350116 7684 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:10.350584 7685 heartbeater.cc:511] Master 127.0.97.61:43389 requested a full tablet report, sending...
I20260923 20:54:10.350656 7684 heartbeater.cc:511] Master 127.0.97.62:43651 requested a full tablet report, sending...
I20260923 20:54:10.351507 7021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:10.352130 7683 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34141
I20260923 20:54:10.352264 7683 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:10.352381 6569 ts_manager.cc:194] Re-registered known tserver with Master: dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083)
I20260923 20:54:10.352684 7683 heartbeater.cc:511] Master 127.0.97.60:34141 requested a full tablet report, sending...
I20260923 20:54:10.353549 6569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41775
I20260923 20:54:10.353861 7359 ts_manager.cc:194] Registered new tserver with Master: dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083)
I20260923 20:54:10.355023 7170 ts_manager.cc:194] Registered new tserver with Master: dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083)
I20260923 20:54:10.445948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6898
I20260923 20:54:10.461987 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34533
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43651
--webserver_port=45745
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:54:10.700695 7538 heartbeater.cc:503] Master 127.0.97.62:43651 was elected leader, sending a full tablet report...
W20260923 20:54:10.819005 7696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:10.819403 7696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:10.819581 7696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:10.827764 7696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:10.827909 7696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:10.828030 7696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:10.838299 7696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45745
--tserver_master_addrs=127.0.97.62:43651,127.0.97.61:43389,127.0.97.60:34141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:10.839641 7696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:10.840950 7696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:10.856247 7701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:10.856290 7702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:10.856527 7704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:10.857570 7696 server_base.cc:1034] running on GCE node
I20260923 20:54:10.858326 7696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:10.859386 7696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:10.860637 7696 hybrid_clock.cc:648] HybridClock initialized: now 1790196850860588 us; error 44 us; skew 500 ppm
I20260923 20:54:10.861042 7696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:10.864530 7696 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 20:54:10.866204 7696 webserver.cc:460] Webserver started at http://127.0.97.3:45745/ using document root <none> and password file <none>
I20260923 20:54:10.866971 7696 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:10.867141 7696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:10.873389 7696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 20:54:10.876977 7711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:10.878824 7696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:54:10.879045 7696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "8ee911db158c4ca6bf09fee02e19da8b"
format_stamp: "Formatted at 2026-09-23 20:54:07 on dist-test-slave-plcg"
server_key: "87881f2dfc34bca62d05f19d4b2cf494"
server_key_iv: "7f8bc03a5771cba2cd3a0806766b77b9"
server_key_version: "encryptionkey@0"
I20260923 20:54:10.880151 7696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:10.901340 7696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:10.906457 7696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:10.906795 7696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:10.907843 7696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:10.909339 7696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:10.909473 7696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:10.909621 7696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:10.909718 7696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:10.946156 7696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34533
I20260923 20:54:10.946259 7823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34533 every 8 connection(s)
I20260923 20:54:10.947644 7696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:10.955507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 7696
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:10.997331 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.951603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.97.3:39667)
0923 20:54:10.966109 (+ 14506us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:10.966136 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:10.966311 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:10.971591 (+ 5280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:10.971610 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:10.972988 (+ 1378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:10.974559 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.974599 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.976617 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.976622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.979021 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.979036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.980338 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.980345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.980671 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.983209 (+ 2538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:10.983243 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:10.987341 (+ 4098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:10.991516 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.991526 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.991550 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.992569 (+ 1019us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.993086 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.993094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.993099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.993393 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.994218 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:10.994227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:10.994718 (+ 491us) client_negotiation.cc:769] Sending connection context
0923 20:54:10.995070 (+ 352us) client_negotiation.cc:240] Negotiation successful
0923 20:54:10.995677 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14295,"mutex_wait_us":433,"thread_start_us":13647,"threads_started":1}
Sep 23 20:54:10 dist-test-slave-plcg krb5kdc[6524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196850, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 20:54:10.998901 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.950834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39667 (local address 127.0.97.61:43389)
0923 20:54:10.970600 (+ 19766us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.970606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.970627 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:54:10.970698 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.970703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.970778 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.970896 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.974894 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.976377 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.979229 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.979887 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.980026 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.987588 (+ 7562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:10.987596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:10.987601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:10.987639 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:10.991321 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.992752 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.992759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.992765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.992932 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.993557 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.993564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.993570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.993990 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:10.994142 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:10.998183 (+ 4041us) server_negotiation.cc:299] Negotiation successful
0923 20:54:10.998447 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19630,"thread_start_us":19701,"threads_started":1}
I20260923 20:54:11.000427 7826 heartbeater.cc:347] Connected to a master server at 127.0.97.61:43389
I20260923 20:54:11.000833 7826 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:11.001963 7826 heartbeater.cc:511] Master 127.0.97.61:43389 requested a full tablet report, sending...
I20260923 20:54:11.002719 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.951601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:40497)
0923 20:54:10.961848 (+ 10247us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:10.961911 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:10.965062 (+ 3151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:10.975879 (+ 10817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:10.975885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:10.976466 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:10.976804 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.976815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.981200 (+ 4385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.981207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.982894 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.982908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.984870 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.984877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.985048 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.987607 (+ 2559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:10.987631 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:10.990844 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:10.997511 (+ 6667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.997516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.997520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.998053 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:10.998573 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:10.998578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:10.998584 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:10.998772 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:11.001426 (+ 2654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:11.001433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:11.001639 (+ 206us) client_negotiation.cc:769] Sending connection context
0923 20:54:11.001811 (+ 172us) client_negotiation.cc:240] Negotiation successful
0923 20:54:11.002170 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9859,"mutex_wait_us":4,"thread_start_us":9560,"threads_started":1}
I20260923 20:54:11.003898 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.950930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40497 (local address 127.0.97.62:43651)
0923 20:54:10.952428 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.952433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.974313 (+ 21880us) server_negotiation.cc:407] Connection header received
0923 20:54:10.974729 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.974739 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.975362 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.975856 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.976950 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.979770 (+ 2820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.983048 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.984461 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.984624 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.991131 (+ 6507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:10.991140 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:10.991146 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:10.991194 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:10.994978 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.998214 (+ 3236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.998220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.998224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.998428 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:10.998911 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:10.998919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:10.998925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:10.999301 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:10.999451 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.003136 (+ 3685us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.003461 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":1261,"threads_started":1}
I20260923 20:54:11.004453 7170 ts_manager.cc:194] Registered new tserver with Master: 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533)
I20260923 20:54:11.004725 7825 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43651
I20260923 20:54:11.004904 7825 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:11.005453 7825 heartbeater.cc:511] Master 127.0.97.62:43651 requested a full tablet report, sending...
I20260923 20:54:11.006608 6569 ts_manager.cc:194] Re-registered known tserver with Master: 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533)
I20260923 20:54:11.007962 6569 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40497
I20260923 20:54:11.010329 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.962769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.97.3:38987)
0923 20:54:10.970324 (+ 7555us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:10.970348 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:10.970479 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:10.980733 (+ 10254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:10.980737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:10.981446 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:10.981872 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.981886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.985580 (+ 3694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.985588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.987163 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:10.987179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.992056 (+ 4877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.992067 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:10.995338 (+ 3271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:10.995962 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:10.995983 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:11.000488 (+ 4505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:11.007057 (+ 6569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:11.007063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:11.007067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:11.007627 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:11.008160 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:11.008164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:11.008168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:11.008350 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:11.009148 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:11.009161 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:11.009330 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 20:54:11.009512 (+ 182us) client_negotiation.cc:240] Negotiation successful
0923 20:54:11.009883 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7416,"mutex_wait_us":42,"thread_start_us":7255,"threads_started":1}
I20260923 20:54:11.012286 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.962756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38987 (local address 127.0.97.60:34141)
0923 20:54:10.980258 (+ 17502us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.980267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.980288 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:54:10.980347 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.980351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.980450 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.980564 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.982071 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.985442 (+ 3371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.987363 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.989749 (+ 2386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.989960 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.000754 (+ 10794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.000760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.000764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.000801 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.006843 (+ 6042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.007812 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.007818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.007822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.008001 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.008487 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.008493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.008498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.008923 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.009070 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.011646 (+ 2576us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.011847 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17354,"thread_start_us":17257,"threads_started":1}
I20260923 20:54:11.013269 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.959785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.0.97.62:43651)
0923 20:54:10.986621 (+ 26836us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.986631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:10.986664 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:54:10.986736 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:10.986745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:10.986866 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:10.987022 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:10.992074 (+ 5052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.994583 (+ 2509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.997852 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:10.998570 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:10.998694 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.003317 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.003328 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.003337 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.003381 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.008338 (+ 4957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.009661 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.009672 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.009678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.009915 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.011245 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.011268 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.011275 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.011769 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.012127 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.012558 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.012827 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26530,"thread_start_us":26433,"threads_started":1}
I20260923 20:54:11.013674 7824 heartbeater.cc:347] Connected to a master server at 127.0.97.60:34141
I20260923 20:54:11.014300 7824 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:11.014822 7824 heartbeater.cc:511] Master 127.0.97.60:34141 requested a full tablet report, sending...
I20260923 20:54:11.016111 7359 ts_manager.cc:194] Registered new tserver with Master: 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533)
I20260923 20:54:11.019327 6569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:54:11.029634 6569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:11.031816 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.970719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.97.61:43389)
0923 20:54:10.978468 (+ 7749us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:10.978474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.012765 (+ 34291us) server_negotiation.cc:407] Connection header received
0923 20:54:11.012840 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.012844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.012941 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.013063 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:11.016772 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.018059 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.020056 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.020730 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.020895 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.024977 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.024985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.024990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.025038 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.028638 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.029449 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.029455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.029459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.029614 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.030124 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.030129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.030137 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.030422 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.030685 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.031053 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.031279 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7606,"thread_start_us":7765,"threads_started":1}
I20260923 20:54:11.033049 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:10.990795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.97.60:34141)
0923 20:54:11.012562 (+ 21767us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.012566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.012581 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:54:11.012625 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.012629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.012741 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.012857 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:11.017897 (+ 5040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.018904 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.020562 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.021197 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.021753 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.026010 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.026018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.026023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.026121 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.029665 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.030666 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.030672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.030679 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.030862 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.031326 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.031331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.031334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.031652 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.031903 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.032283 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.032510 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21655,"thread_start_us":21530,"threads_started":1}
I20260923 20:54:11.076165 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.055895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.0.97.3:34533)
0923 20:54:11.069072 (+ 13177us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.069079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.069110 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:54:11.069194 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.069200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.069423 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.069665 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:11.070327 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.071445 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.073475 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.075044 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.075204 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.075500 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.075652 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.075835 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13027,"thread_start_us":12829,"threads_started":1}
I20260923 20:54:11.076231 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.055706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34533 (local address 127.0.0.1:49758)
0923 20:54:11.062846 (+ 7140us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:11.062878 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:11.063070 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:11.069650 (+ 6580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:11.069656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:11.069685 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:11.070112 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.070123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.071610 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.071616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.073266 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.073280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.075271 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.075278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.075455 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.075493 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:54:11.075610 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 20:54:11.075798 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":433,"threads_started":1}
I20260923 20:54:11.088872 7759 tablet_service.cc:1467] Processing CreateTablet for tablet f8a9f55c79ea4d018ef13f2fe4ac6906 (DEFAULT_TABLE table=test-table [id=09c9c4ce7df24de4a33c908cfc13eefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:11.089785 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.055897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41167 (local address 127.0.0.1:33708)
0923 20:54:11.076482 (+ 20585us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:11.076502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:11.076612 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:11.082874 (+ 6262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:11.082879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:11.082908 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:11.083117 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.083133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.085624 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.085630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.087393 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.087413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.088969 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.088975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.089148 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.089184 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:54:11.089294 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:54:11.089452 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20488,"thread_start_us":27363,"threads_started":1}
I20260923 20:54:11.090212 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.056437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.97.1:41167)
0923 20:54:11.082271 (+ 25834us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.082278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.082305 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:54:11.082392 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.082398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.082621 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.082872 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:11.083344 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.085490 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.087600 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.088785 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.088955 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.089173 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.089360 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.089526 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25675,"thread_start_us":25558,"threads_started":1}
I20260923 20:54:11.090688 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a9f55c79ea4d018ef13f2fe4ac6906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:11.092070 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.056882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.0.97.2:38083)
0923 20:54:11.068269 (+ 11387us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.068283 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.083719 (+ 15436us) server_negotiation.cc:407] Connection header received
0923 20:54:11.083917 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.083929 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.084233 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.084466 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:11.085176 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.086718 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.089186 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.090601 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.090757 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.090961 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.091581 (+ 620us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.091776 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11132,"thread_start_us":11397,"threads_started":1}
I20260923 20:54:11.092890 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.056501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38083 (local address 127.0.0.1:40264)
0923 20:54:11.083563 (+ 27062us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:11.083585 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:11.083714 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:11.084474 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:11.084478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:11.084506 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:11.084876 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.084982 (+ 106us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.086921 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.086927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.088984 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.089003 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.091157 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.091163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.091358 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.091392 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:54:11.092350 (+ 958us) client_negotiation.cc:240] Negotiation successful
0923 20:54:11.092511 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26912,"thread_start_us":27512,"threads_started":1}
I20260923 20:54:11.093860 7472 tablet_service.cc:1467] Processing CreateTablet for tablet f8a9f55c79ea4d018ef13f2fe4ac6906 (DEFAULT_TABLE table=test-table [id=09c9c4ce7df24de4a33c908cfc13eefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:11.095163 7472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a9f55c79ea4d018ef13f2fe4ac6906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:11.096905 7618 tablet_service.cc:1467] Processing CreateTablet for tablet f8a9f55c79ea4d018ef13f2fe4ac6906 (DEFAULT_TABLE table=test-table [id=09c9c4ce7df24de4a33c908cfc13eefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:11.098412 7618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a9f55c79ea4d018ef13f2fe4ac6906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:11.109251 7861 tablet_bootstrap.cc:492] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Bootstrap starting.
I20260923 20:54:11.110074 7860 tablet_bootstrap.cc:492] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Bootstrap starting.
I20260923 20:54:11.112804 7860 tablet_bootstrap.cc:654] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:11.112964 7861 tablet_bootstrap.cc:654] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:11.114125 7860 log.cc:824] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:11.114267 7861 log.cc:824] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:11.117781 7864 tablet_bootstrap.cc:492] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Bootstrap starting.
I20260923 20:54:11.117982 7861 tablet_bootstrap.cc:492] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: No bootstrap required, opened a new log
I20260923 20:54:11.118011 7860 tablet_bootstrap.cc:492] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: No bootstrap required, opened a new log
I20260923 20:54:11.118269 7861 ts_tablet_manager.cc:1397] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.007s
I20260923 20:54:11.118287 7860 ts_tablet_manager.cc:1397] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:54:11.121501 7864 tablet_bootstrap.cc:654] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:11.123303 7864 log.cc:824] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:11.127002 7864 tablet_bootstrap.cc:492] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: No bootstrap required, opened a new log
I20260923 20:54:11.127322 7864 ts_tablet_manager.cc:1397] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260923 20:54:11.128247 7861 raft_consensus.cc:348] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.128671 7861 raft_consensus.cc:374] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:11.128774 7861 raft_consensus.cc:729] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16771be618554d49947f6fc449f5053b, State: Initialized, Role: FOLLOWER
I20260923 20:54:11.129434 7861 consensus_queue.cc:260] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.129326 7860 raft_consensus.cc:348] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.129737 7860 raft_consensus.cc:374] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:11.129875 7860 raft_consensus.cc:729] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee911db158c4ca6bf09fee02e19da8b, State: Initialized, Role: FOLLOWER
I20260923 20:54:11.130558 7860 consensus_queue.cc:260] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.132891 7860 ts_tablet_manager.cc:1428] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260923 20:54:11.133333 7825 heartbeater.cc:503] Master 127.0.97.62:43651 was elected leader, sending a full tablet report...
I20260923 20:54:11.138408 7861 ts_tablet_manager.cc:1428] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
I20260923 20:54:11.138578 7864 raft_consensus.cc:348] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.139075 7864 raft_consensus.cc:374] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:11.139184 7864 raft_consensus.cc:729] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dece15634329499a8b2456f9ceae3f0d, State: Initialized, Role: FOLLOWER
I20260923 20:54:11.139844 7864 consensus_queue.cc:260] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.144379 7864 ts_tablet_manager.cc:1428] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260923 20:54:11.144690 7684 heartbeater.cc:503] Master 127.0.97.62:43651 was elected leader, sending a full tablet report...
I20260923 20:54:11.148510 7868 raft_consensus.cc:482] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:11.148818 7868 raft_consensus.cc:504] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.151122 7868 leader_election.cc:288] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533), 16771be618554d49947f6fc449f5053b (127.0.97.1:41167)
W20260923 20:54:11.153645 7540 tablet.cc:2369] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:54:11.159504 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.151861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48439 (local address 127.0.97.1:41167)
0923 20:54:11.152128 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.152137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.152771 (+ 634us) server_negotiation.cc:407] Connection header received
0923 20:54:11.152986 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.152991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.153060 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.153173 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:11.153705 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.154825 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.157175 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.158469 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.158602 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.158717 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.159126 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.159260 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 20:54:11.159859 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.151387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41167 (local address 127.0.97.2:48439)
0923 20:54:11.152654 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:11.152670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:11.152845 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:11.153187 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:11.153191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:11.153215 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:11.153563 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.153572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.154974 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.154978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.157016 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.157029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.158709 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.158718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.158900 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.158937 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:54:11.159035 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 20:54:11.159189 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":2498,"threads_started":1}
I20260923 20:54:11.160981 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.151428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34533 (local address 127.0.97.2:37193)
0923 20:54:11.154152 (+ 2724us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:11.154171 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:11.154342 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:11.154736 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:11.154741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:11.154775 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:11.155142 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.155154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.156696 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.156704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.158799 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:11.158813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.160352 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.160360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:11.160533 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.160556 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:54:11.160633 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:54:11.160722 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2597,"mutex_wait_us":1,"thread_start_us":902,"threads_started":1}
I20260923 20:54:11.161584 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.151640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37193 (local address 127.0.97.3:34533)
0923 20:54:11.151925 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.151934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.154249 (+ 2315us) server_negotiation.cc:407] Connection header received
0923 20:54:11.154518 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.154523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.154606 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.154690 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:11.155309 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.156444 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.158965 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.160145 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.160266 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.160341 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.160738 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.160882 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 20:54:11.161088 7492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9f55c79ea4d018ef13f2fe4ac6906" candidate_uuid: "dece15634329499a8b2456f9ceae3f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16771be618554d49947f6fc449f5053b" is_pre_election: true
I20260923 20:54:11.162119 7492 raft_consensus.cc:2454] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dece15634329499a8b2456f9ceae3f0d in term 0.
I20260923 20:54:11.162587 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9f55c79ea4d018ef13f2fe4ac6906" candidate_uuid: "dece15634329499a8b2456f9ceae3f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee911db158c4ca6bf09fee02e19da8b" is_pre_election: true
I20260923 20:54:11.162904 7573 leader_election.cc:302] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16771be618554d49947f6fc449f5053b, dece15634329499a8b2456f9ceae3f0d; no voters:
I20260923 20:54:11.163064 7779 raft_consensus.cc:2454] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dece15634329499a8b2456f9ceae3f0d in term 0.
I20260923 20:54:11.163367 7868 raft_consensus.cc:2781] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:11.163522 7868 raft_consensus.cc:482] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:11.163601 7868 raft_consensus.cc:3037] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:11.166056 7868 raft_consensus.cc:504] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.167251 7868 leader_election.cc:288] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [CANDIDATE]: Term 1 election: Requested vote from peers 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533), 16771be618554d49947f6fc449f5053b (127.0.97.1:41167)
I20260923 20:54:11.168170 7779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9f55c79ea4d018ef13f2fe4ac6906" candidate_uuid: "dece15634329499a8b2456f9ceae3f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee911db158c4ca6bf09fee02e19da8b"
I20260923 20:54:11.168350 7492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a9f55c79ea4d018ef13f2fe4ac6906" candidate_uuid: "dece15634329499a8b2456f9ceae3f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16771be618554d49947f6fc449f5053b"
I20260923 20:54:11.168460 7779 raft_consensus.cc:3037] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:11.168637 7492 raft_consensus.cc:3037] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:11.170986 7492 raft_consensus.cc:2454] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dece15634329499a8b2456f9ceae3f0d in term 1.
I20260923 20:54:11.170987 7779 raft_consensus.cc:2454] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dece15634329499a8b2456f9ceae3f0d in term 1.
I20260923 20:54:11.171564 7573 leader_election.cc:302] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16771be618554d49947f6fc449f5053b, 8ee911db158c4ca6bf09fee02e19da8b, dece15634329499a8b2456f9ceae3f0d; no voters:
I20260923 20:54:11.172003 7868 raft_consensus.cc:2781] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:11.172510 7868 raft_consensus.cc:686] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 1 LEADER]: Becoming Leader. State: Replica: dece15634329499a8b2456f9ceae3f0d, State: Running, Role: LEADER
I20260923 20:54:11.173136 7868 consensus_queue.cc:237] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } }
I20260923 20:54:11.179179 6568 catalog_manager.cc:5495] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d reported cstate change: term changed from 0 to 1, leader changed from <none> to dece15634329499a8b2456f9ceae3f0d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "dece15634329499a8b2456f9ceae3f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } health_report { overall_health: UNKNOWN } } }
W20260923 20:54:11.202428 7827 tablet.cc:2369] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:54:11.230002 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.223510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.0.97.2:38083)
0923 20:54:11.223750 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.223756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.223920 (+ 164us) server_negotiation.cc:407] Connection header received
0923 20:54:11.224144 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.224151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.224254 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.224428 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:11.224924 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.226006 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.227393 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.227794 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.227893 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.228384 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:11.228928 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:11.229036 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.229155 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.229225 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:54:11.239212 7492 raft_consensus.cc:1261] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Refusing update from remote peer dece15634329499a8b2456f9ceae3f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:11.239208 7779 raft_consensus.cc:1261] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 FOLLOWER]: Refusing update from remote peer dece15634329499a8b2456f9ceae3f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:11.240339 7871 consensus_queue.cc:1035] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:11.240716 7868 consensus_queue.cc:1035] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:11.254020 7875 mvcc.cc:205] Tried to move back new op lower bound from 7332646302665072640 to 7332646302409871360. Current Snapshot: MvccSnapshot[applied={T|T < 7332646302665072640}]
I20260923 20:54:11.262953 7877 mvcc.cc:205] Tried to move back new op lower bound from 7332646302665072640 to 7332646302409871360. Current Snapshot: MvccSnapshot[applied={T|T < 7332646302665072640}]
I20260923 20:54:11.282620 6568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:11.282953 6568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:11.290867 6568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d08e7f6a2bf48c888ae4860125d0912: Sending DeleteTablet for 3 replicas of tablet f8a9f55c79ea4d018ef13f2fe4ac6906
I20260923 20:54:11.292243 7759 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a9f55c79ea4d018ef13f2fe4ac6906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49758
I20260923 20:54:11.292416 7618 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a9f55c79ea4d018ef13f2fe4ac6906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40264
I20260923 20:54:11.292464 7472 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a9f55c79ea4d018ef13f2fe4ac6906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33708
I20260923 20:54:11.303053 7886 tablet_replica.cc:330] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: stopping tablet replica
I20260923 20:54:11.303756 7886 raft_consensus.cc:2229] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:54:11.304394 7884 tablet_replica.cc:330] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: stopping tablet replica
I20260923 20:54:11.304579 7886 raft_consensus.cc:2258] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:11.304811 7884 raft_consensus.cc:2229] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:11.305308 7884 raft_consensus.cc:2258] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:11.307576 7884 ts_tablet_manager.cc:1905] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:11.311523 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 6537
I20260923 20:54:11.312783 6599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:11.323904 7884 ts_tablet_manager.cc:1918] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:11.324083 7884 log.cc:1196] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/f8a9f55c79ea4d018ef13f2fe4ac6906
I20260923 20:54:11.324616 7884 ts_tablet_manager.cc:1939] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 16771be618554d49947f6fc449f5053b: Deleting consensus metadata
I20260923 20:54:11.326606 7885 tablet_replica.cc:330] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: stopping tablet replica
I20260923 20:54:11.327904 6555 catalog_manager.cc:4841] TS 16771be618554d49947f6fc449f5053b (127.0.97.1:41167): tablet f8a9f55c79ea4d018ef13f2fe4ac6906 (table test-table [id=09c9c4ce7df24de4a33c908cfc13eefe]) successfully deleted
I20260923 20:54:11.328503 7886 ts_tablet_manager.cc:1905] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:11.328510 7885 raft_consensus.cc:2229] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:11.329526 7885 raft_consensus.cc:2258] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:11.332194 7885 ts_tablet_manager.cc:1905] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:11.341713 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.295862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.0.97.61:43389)
0923 20:54:11.296126 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.296135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.310507 (+ 14372us) server_negotiation.cc:407] Connection header received
0923 20:54:11.311033 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.311041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.311160 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.311259 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:11.315262 (+ 4003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.316790 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.319795 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.320737 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.320877 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.326236 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.326243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.326248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.326295 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.331075 (+ 4780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.332037 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.332042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.332046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.332186 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.332744 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.332749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.332753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.333106 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.334877 (+ 1771us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.340545 (+ 5668us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.340872 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:54:11.344815 7885 ts_tablet_manager.cc:1918] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:11.345026 7885 log.cc:1196] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/f8a9f55c79ea4d018ef13f2fe4ac6906
I20260923 20:54:11.345595 7885 ts_tablet_manager.cc:1939] T f8a9f55c79ea4d018ef13f2fe4ac6906 P 8ee911db158c4ca6bf09fee02e19da8b: Deleting consensus metadata
I20260923 20:54:11.348243 7886 ts_tablet_manager.cc:1918] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:11.348438 7886 log.cc:1196] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/f8a9f55c79ea4d018ef13f2fe4ac6906
I20260923 20:54:11.349044 7886 ts_tablet_manager.cc:1939] T f8a9f55c79ea4d018ef13f2fe4ac6906 P dece15634329499a8b2456f9ceae3f0d: Deleting consensus metadata
I20260923 20:54:11.356050 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.310674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.97.60:34141)
0923 20:54:11.310887 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.310894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.334415 (+ 23521us) server_negotiation.cc:407] Connection header received
0923 20:54:11.334563 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.334570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.334663 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.334781 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:11.337261 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.338414 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.342709 (+ 4295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.343402 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.343519 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.347987 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.347997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.348002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.348049 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.352029 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.353412 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.353419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.353423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.353597 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.354148 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.354153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.354157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.354532 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.354774 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.355431 (+ 657us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.355686 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:54:11.462011 6556 catalog_manager.cc:4841] TS dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083): tablet f8a9f55c79ea4d018ef13f2fe4ac6906 (table test-table [id=09c9c4ce7df24de4a33c908cfc13eefe]) successfully deleted
I20260923 20:54:11.462363 6556 catalog_manager.cc:4841] TS 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533): tablet f8a9f55c79ea4d018ef13f2fe4ac6906 (table test-table [id=09c9c4ce7df24de4a33c908cfc13eefe]) successfully deleted
I20260923 20:54:11.462632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 6537
I20260923 20:54:11.507624 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.490165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.0.97.61:43389)
0923 20:54:11.490373 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.490381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.490982 (+ 601us) server_negotiation.cc:407] Connection header received
0923 20:54:11.491285 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.491292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.491401 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.491476 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:11.494517 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.495462 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.497134 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.497857 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.497994 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.501300 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.501306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.501309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.501348 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.504692 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.505444 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.505451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.505454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.505587 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.506043 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.506048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.506052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.506438 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.506681 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.507053 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.507276 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:54:11.509119 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:11.491967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.97.60:34141)
0923 20:54:11.492173 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:11.492180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:11.492198 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:54:11.492265 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:11.492273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:11.492384 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:11.492489 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:11.494997 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.496111 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.497714 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:11.498411 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:11.498550 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:11.502057 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:11.502064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:11.502069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:11.502113 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:11.505800 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.506460 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.506467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.506470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.506615 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:11.507060 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:11.507065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:11.507068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:11.507447 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:11.507671 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:11.508496 (+ 825us) server_negotiation.cc:299] Negotiation successful
0923 20:54:11.508727 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:54:12.269871 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.268835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.2:43643)
0923 20:54:12.269505 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.269640 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":450,"threads_started":1}
W20260923 20:54:12.270355 7684 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43651 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.271206 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.270492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.1:53945)
0923 20:54:12.270958 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.271055 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":255,"threads_started":1}
W20260923 20:54:12.271564 7538 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43651 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.277289 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.276476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:50819)
0923 20:54:12.276952 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.277115 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":284,"threads_started":1}
W20260923 20:54:12.277627 7825 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43651 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.790374 7899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d08e7f6a2bf48c888ae4860125d0912)
I20260923 20:54:12.790589 7900 raft_consensus.cc:482] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d08e7f6a2bf48c888ae4860125d0912)
I20260923 20:54:12.790726 7899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } }
I20260923 20:54:12.790840 7900 raft_consensus.cc:504] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } }
I20260923 20:54:12.792119 7900 leader_election.cc:288] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d08e7f6a2bf48c888ae4860125d0912 (127.0.97.62:43651), 3c52ce98709249a5912a4ef7e67c236c (127.0.97.60:34141)
I20260923 20:54:12.793454 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.792246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45578)
0923 20:54:12.792866 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.793104 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":523,"threads_started":1}
W20260923 20:54:12.793954 7344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.798223 7899 leader_election.cc:288] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d08e7f6a2bf48c888ae4860125d0912 (127.0.97.62:43651), e03b196350184da591531013dfe003d0 (127.0.97.61:43389)
I20260923 20:54:12.804692 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.792237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45568)
0923 20:54:12.804123 (+ 11886us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.804345 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11732,"thread_start_us":11575,"threads_started":1}
W20260923 20:54:12.805416 7156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.806950 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.806326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45594)
0923 20:54:12.806577 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.806701 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260923 20:54:12.807171 7344 leader_election.cc:334] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d08e7f6a2bf48c888ae4860125d0912 (127.0.97.62:43651): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.808871 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.807962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45598)
0923 20:54:12.808523 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.808658 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":311,"threads_started":1}
W20260923 20:54:12.809123 7156 leader_election.cc:334] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d08e7f6a2bf48c888ae4860125d0912 (127.0.97.62:43651): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.811405 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.792612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.97.61:43389)
0923 20:54:12.804324 (+ 11712us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.804331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.804358 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:54:12.804437 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.804442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.804526 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.804676 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:12.805197 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.806729 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.809430 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.810371 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.810529 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.810622 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:12.810898 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 20:54:12.811028 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":11591,"thread_start_us":11369,"threads_started":1}
I20260923 20:54:12.811446 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.792269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43389 (local address 127.0.0.1:43330)
0923 20:54:12.798909 (+ 6640us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.798941 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:12.799145 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:12.804695 (+ 5550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:12.804700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:12.804732 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:12.804994 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.805008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.806966 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.806975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.809256 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.809292 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.810557 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.810563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.810733 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.810758 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:54:12.810868 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:54:12.811026 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6474,"mutex_wait_us":22,"thread_start_us":11358,"threads_started":1}
I20260923 20:54:12.812041 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c52ce98709249a5912a4ef7e67c236c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e03b196350184da591531013dfe003d0" is_pre_election: true
I20260923 20:54:12.812403 7179 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c52ce98709249a5912a4ef7e67c236c in term 1.
I20260923 20:54:12.812927 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.805071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34141 (local address 127.0.0.1:32956)
0923 20:54:12.805256 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.805284 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:12.805454 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:12.806337 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:12.806342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:12.806371 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:12.806766 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.806779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.809126 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.809132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.810773 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.810786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.812166 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.812172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.812339 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.812361 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:54:12.812470 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:54:12.812617 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":54}
I20260923 20:54:12.813023 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.804482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.0.97.60:34141)
0923 20:54:12.806012 (+ 1530us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.806017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.806039 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:54:12.806098 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.806102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.806173 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.806914 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:12.807726 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.808943 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.810932 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.811973 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.812114 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.812224 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:12.812512 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 20:54:12.812647 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":1399,"thread_start_us":1567,"threads_started":1}
I20260923 20:54:12.812940 7342 leader_election.cc:302] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c52ce98709249a5912a4ef7e67c236c, e03b196350184da591531013dfe003d0; no voters: 6d08e7f6a2bf48c888ae4860125d0912
I20260923 20:54:12.813463 7899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 20:54:12.813622 7899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d08e7f6a2bf48c888ae4860125d0912)
I20260923 20:54:12.813735 7899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:54:12.813786 7369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e03b196350184da591531013dfe003d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c52ce98709249a5912a4ef7e67c236c" is_pre_election: true
I20260923 20:54:12.816093 7899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } }
I20260923 20:54:12.816557 7369 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e03b196350184da591531013dfe003d0 in current term 2: Already voted for candidate 3c52ce98709249a5912a4ef7e67c236c in this term.
I20260923 20:54:12.816954 7899 leader_election.cc:288] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [CANDIDATE]: Term 2 election: Requested vote from peers 6d08e7f6a2bf48c888ae4860125d0912 (127.0.97.62:43651), e03b196350184da591531013dfe003d0 (127.0.97.61:43389)
I20260923 20:54:12.817443 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.817019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45600)
0923 20:54:12.817208 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.817299 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:54:12.817637 7179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c52ce98709249a5912a4ef7e67c236c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e03b196350184da591531013dfe003d0"
I20260923 20:54:12.817905 7179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:54:12.820211 7156 leader_election.cc:302] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e03b196350184da591531013dfe003d0; no voters: 3c52ce98709249a5912a4ef7e67c236c, 6d08e7f6a2bf48c888ae4860125d0912
I20260923 20:54:12.820365 7179 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c52ce98709249a5912a4ef7e67c236c in term 2.
I20260923 20:54:12.820643 7900 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:54:12.820758 7342 leader_election.cc:302] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c52ce98709249a5912a4ef7e67c236c, e03b196350184da591531013dfe003d0; no voters:
I20260923 20:54:12.820825 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.820332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45614)
0923 20:54:12.820560 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.820658 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260923 20:54:12.820977 7344 leader_election.cc:334] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6d08e7f6a2bf48c888ae4860125d0912 (127.0.97.62:43651): Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
I20260923 20:54:12.821050 7899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:54:12.821519 7899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [term 2 LEADER]: Becoming Leader. State: Replica: 3c52ce98709249a5912a4ef7e67c236c, State: Running, Role: LEADER
I20260923 20:54:12.821954 7899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } }
I20260923 20:54:12.823910 7909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c52ce98709249a5912a4ef7e67c236c. Latest consensus state: current_term: 2 leader_uuid: "3c52ce98709249a5912a4ef7e67c236c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:12.824375 7909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:12.825017 7911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:12.828805 7911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:12.829623 7911 catalog_manager.cc:1259] Loaded cluster ID: bebeaef8373045cbba5fa1858e3cfb24
I20260923 20:54:12.829697 7911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:12.830864 7911 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:12.831624 7911 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Loaded TSK: 0
I20260923 20:54:12.832427 7911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:54:12.848125 7358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:54:12.855551 7358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:12.860174 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.859693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45620)
0923 20:54:12.859922 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.860013 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:54:12.860198 7179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [term 2 FOLLOWER]: Refusing update from remote peer 3c52ce98709249a5912a4ef7e67c236c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 20:54:12.861045 7909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 20:54:12.861339 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.860928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45630)
0923 20:54:12.861085 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.861181 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:54:12.861495 7344 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c -> Peer 6d08e7f6a2bf48c888ae4860125d0912 (127.0.97.62:43651): Couldn't send request to peer 6d08e7f6a2bf48c888ae4860125d0912. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:54:12.864619 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c52ce98709249a5912a4ef7e67c236c. Latest consensus state: current_term: 2 leader_uuid: "3c52ce98709249a5912a4ef7e67c236c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:12.864966 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:54:12.865753 7909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3c52ce98709249a5912a4ef7e67c236c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:12.866115 7909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:12.867303 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3c52ce98709249a5912a4ef7e67c236c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6d08e7f6a2bf48c888ae4860125d0912" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43651 } } peers { permanent_uuid: "e03b196350184da591531013dfe003d0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43389 } attrs { promote: false } } peers { permanent_uuid: "3c52ce98709249a5912a4ef7e67c236c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34141 } attrs { promote: false } } }
I20260923 20:54:12.867561 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03b196350184da591531013dfe003d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:54:12.895661 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.875644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38083 (local address 127.0.0.1:40270)
0923 20:54:12.876078 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.876100 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:12.876224 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:12.884965 (+ 8741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:12.884976 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:12.885024 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:12.885513 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.885530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.890874 (+ 5344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.890883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.893003 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.893016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.894690 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.894700 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.894964 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.895024 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 20:54:12.895109 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 20:54:12.895321 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300}
I20260923 20:54:12.896474 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.876468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.97.2:38083)
0923 20:54:12.883458 (+ 6990us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.883469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.883504 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:54:12.883609 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.883618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.884473 (+ 855us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.885778 (+ 1305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:12.886166 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.888586 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.893225 (+ 4639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.894356 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.896038 (+ 1682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.896112 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:12.896154 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 20:54:12.896254 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6784,"thread_start_us":6615,"threads_started":1}
I20260923 20:54:12.896659 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.875225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41167 (local address 127.0.0.1:33722)
0923 20:54:12.875399 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.875421 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:12.875532 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:12.891171 (+ 15639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:12.891178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:12.891213 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:12.891495 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.891505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.892885 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.892889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.894316 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.894350 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.895868 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.895874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.896066 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.896104 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:54:12.896163 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 20:54:12.896333 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 20:54:12.897167 7618 tablet_service.cc:1467] Processing CreateTablet for tablet 7f483e6d93a448bcae05c8821ebe8e29 (DEFAULT_TABLE table=test-table [id=0a8d7c8e174548388ece141804e9210f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:12.897600 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.875300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.97.1:41167)
0923 20:54:12.890799 (+ 15499us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.890807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.890828 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:54:12.890896 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.890901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.891001 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.891123 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:12.891653 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.892717 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.894527 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.895696 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.897012 (+ 1316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.897121 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:12.897173 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 20:54:12.897312 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15326,"thread_start_us":15926,"threads_started":1}
I20260923 20:54:12.897614 7618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f483e6d93a448bcae05c8821ebe8e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:12.898487 7472 tablet_service.cc:1467] Processing CreateTablet for tablet 7f483e6d93a448bcae05c8821ebe8e29 (DEFAULT_TABLE table=test-table [id=0a8d7c8e174548388ece141804e9210f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:12.898910 7472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f483e6d93a448bcae05c8821ebe8e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:12.901930 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.877066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.0.97.3:34533)
0923 20:54:12.895641 (+ 18575us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.895649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.895670 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:54:12.895739 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.895747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.895861 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.895955 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:12.896690 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.898080 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.900100 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.900858 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.901013 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.901119 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:12.901440 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 20:54:12.901582 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18448,"thread_start_us":18339,"threads_started":1}
I20260923 20:54:12.901933 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.876921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34533 (local address 127.0.0.1:49764)
0923 20:54:12.891157 (+ 14236us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.891178 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:12.891312 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:12.896131 (+ 4819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:12.896137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:12.896165 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:12.896504 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.896515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.898240 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.898245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.899911 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.899925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.901060 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.901066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.901253 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.901295 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:54:12.901396 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:54:12.901537 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14073,"thread_start_us":14427,"threads_started":1}
I20260923 20:54:12.903123 7759 tablet_service.cc:1467] Processing CreateTablet for tablet 7f483e6d93a448bcae05c8821ebe8e29 (DEFAULT_TABLE table=test-table [id=0a8d7c8e174548388ece141804e9210f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:12.903595 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f483e6d93a448bcae05c8821ebe8e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:12.904560 7922 tablet_bootstrap.cc:492] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: Bootstrap starting.
I20260923 20:54:12.905894 7923 tablet_bootstrap.cc:492] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: Bootstrap starting.
I20260923 20:54:12.906399 7922 tablet_bootstrap.cc:654] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:12.907258 7923 tablet_bootstrap.cc:654] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:12.909145 7922 tablet_bootstrap.cc:492] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: No bootstrap required, opened a new log
I20260923 20:54:12.909324 7922 ts_tablet_manager.cc:1397] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 20:54:12.910007 7923 tablet_bootstrap.cc:492] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: No bootstrap required, opened a new log
I20260923 20:54:12.910243 7923 ts_tablet_manager.cc:1397] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260923 20:54:12.910223 7922 raft_consensus.cc:348] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.910455 7926 tablet_bootstrap.cc:492] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: Bootstrap starting.
I20260923 20:54:12.910466 7922 raft_consensus.cc:374] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:12.910573 7922 raft_consensus.cc:729] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dece15634329499a8b2456f9ceae3f0d, State: Initialized, Role: FOLLOWER
I20260923 20:54:12.910845 7922 consensus_queue.cc:260] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.911322 7923 raft_consensus.cc:348] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.911558 7923 raft_consensus.cc:374] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:12.911659 7923 raft_consensus.cc:729] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16771be618554d49947f6fc449f5053b, State: Initialized, Role: FOLLOWER
I20260923 20:54:12.911680 7922 ts_tablet_manager.cc:1428] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 20:54:12.911921 7923 consensus_queue.cc:260] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.912497 7926 tablet_bootstrap.cc:654] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:12.917323 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.912227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.2:59083)
0923 20:54:12.916885 (+ 4658us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.917027 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4508,"thread_start_us":4387,"threads_started":1}
I20260923 20:54:12.920326 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.919748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.2:39059)
0923 20:54:12.920098 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.920177 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260923 20:54:12.920346 7683 heartbeater.cc:503] Master 127.0.97.60:34141 was elected leader, sending a full tablet report...
I20260923 20:54:12.923126 7923 ts_tablet_manager.cc:1428] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260923 20:54:12.923188 7926 tablet_bootstrap.cc:492] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: No bootstrap required, opened a new log
I20260923 20:54:12.923504 7926 ts_tablet_manager.cc:1397] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260923 20:54:12.924423 7926 raft_consensus.cc:348] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.924688 7926 raft_consensus.cc:374] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:12.924786 7926 raft_consensus.cc:729] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee911db158c4ca6bf09fee02e19da8b, State: Initialized, Role: FOLLOWER
I20260923 20:54:12.925052 7926 consensus_queue.cc:260] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.925621 7537 heartbeater.cc:503] Master 127.0.97.60:34141 was elected leader, sending a full tablet report...
I20260923 20:54:12.925920 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.921218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.1:33551)
0923 20:54:12.924516 (+ 3298us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.925525 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3149,"thread_start_us":2490,"threads_started":1}
I20260923 20:54:12.926048 7926 ts_tablet_manager.cc:1428] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:54:12.927198 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.926724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.1:52543)
0923 20:54:12.926968 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.927055 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 20:54:12.928617 7824 heartbeater.cc:503] Master 127.0.97.60:34141 was elected leader, sending a full tablet report...
I20260923 20:54:12.928925 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.926801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:52375)
0923 20:54:12.928526 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.928632 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1594,"thread_start_us":1422,"threads_started":1}
I20260923 20:54:12.929860 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.929512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:59281)
0923 20:54:12.929642 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.929713 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 20:54:12.959914 7932 raft_consensus.cc:482] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:12.960145 7932 raft_consensus.cc:504] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.961575 7932 leader_election.cc:288] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16771be618554d49947f6fc449f5053b (127.0.97.1:41167), dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083)
I20260923 20:54:12.967779 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.962053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56929 (local address 127.0.97.2:38083)
0923 20:54:12.962224 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.962232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.962381 (+ 149us) server_negotiation.cc:407] Connection header received
0923 20:54:12.962640 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.962646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.962732 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.962829 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:12.963374 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.964161 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.966032 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.966796 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.966895 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.967042 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:12.967373 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 20:54:12.967502 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:54:12.967775 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.961977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47263 (local address 127.0.97.1:41167)
0923 20:54:12.962190 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.962196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.962814 (+ 618us) server_negotiation.cc:407] Connection header received
0923 20:54:12.962958 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.962964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.963047 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.963148 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:12.963646 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.964394 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.966097 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.966817 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.966980 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.967078 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:12.967412 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:54:12.967534 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:54:12.967826 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.961930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38083 (local address 127.0.97.3:56929)
0923 20:54:12.962235 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.962260 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:12.962479 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:12.962897 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:12.962902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:12.962957 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:12.963211 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.963224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.964304 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.964310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.965780 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.965795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.966993 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.966999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.967208 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.967242 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:54:12.967316 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 20:54:12.967491 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":11}
I20260923 20:54:12.968328 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.961942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41167 (local address 127.0.97.3:47263)
0923 20:54:12.962676 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.962692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:12.962831 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:12.963167 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:12.963174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:12.963196 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:12.963502 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.963513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.964531 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.964534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.965936 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:12.965968 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.967045 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:12.967051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:12.967234 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:12.967255 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:54:12.967366 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 20:54:12.967502 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":68,"thread_start_us":363,"threads_started":1}
I20260923 20:54:12.968657 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f483e6d93a448bcae05c8821ebe8e29" candidate_uuid: "8ee911db158c4ca6bf09fee02e19da8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dece15634329499a8b2456f9ceae3f0d" is_pre_election: true
I20260923 20:54:12.968936 7492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f483e6d93a448bcae05c8821ebe8e29" candidate_uuid: "8ee911db158c4ca6bf09fee02e19da8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16771be618554d49947f6fc449f5053b" is_pre_election: true
I20260923 20:54:12.969161 7492 raft_consensus.cc:2454] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee911db158c4ca6bf09fee02e19da8b in term 0.
I20260923 20:54:12.969115 7638 raft_consensus.cc:2454] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee911db158c4ca6bf09fee02e19da8b in term 0.
I20260923 20:54:12.969733 7714 leader_election.cc:302] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16771be618554d49947f6fc449f5053b, 8ee911db158c4ca6bf09fee02e19da8b, dece15634329499a8b2456f9ceae3f0d; no voters:
I20260923 20:54:12.970207 7932 raft_consensus.cc:2781] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:12.970342 7932 raft_consensus.cc:482] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:12.970438 7932 raft_consensus.cc:3037] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:12.972847 7932 raft_consensus.cc:504] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.973385 7932 leader_election.cc:288] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [CANDIDATE]: Term 1 election: Requested vote from peers 16771be618554d49947f6fc449f5053b (127.0.97.1:41167), dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083)
I20260923 20:54:12.973968 7492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f483e6d93a448bcae05c8821ebe8e29" candidate_uuid: "8ee911db158c4ca6bf09fee02e19da8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16771be618554d49947f6fc449f5053b"
I20260923 20:54:12.973994 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f483e6d93a448bcae05c8821ebe8e29" candidate_uuid: "8ee911db158c4ca6bf09fee02e19da8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dece15634329499a8b2456f9ceae3f0d"
I20260923 20:54:12.974193 7638 raft_consensus.cc:3037] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:12.974188 7492 raft_consensus.cc:3037] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:12.976461 7638 raft_consensus.cc:2454] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee911db158c4ca6bf09fee02e19da8b in term 1.
I20260923 20:54:12.976545 7492 raft_consensus.cc:2454] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee911db158c4ca6bf09fee02e19da8b in term 1.
I20260923 20:54:12.976999 7715 leader_election.cc:302] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee911db158c4ca6bf09fee02e19da8b, dece15634329499a8b2456f9ceae3f0d; no voters:
I20260923 20:54:12.977334 7932 raft_consensus.cc:2781] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:12.977807 7932 raft_consensus.cc:686] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 LEADER]: Becoming Leader. State: Replica: 8ee911db158c4ca6bf09fee02e19da8b, State: Running, Role: LEADER
I20260923 20:54:12.978451 7932 consensus_queue.cc:237] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } }
I20260923 20:54:12.978984 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.978494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:42299)
0923 20:54:12.978683 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:12.978813 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:54:12.982275 7358 catalog_manager.cc:5495] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8ee911db158c4ca6bf09fee02e19da8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee911db158c4ca6bf09fee02e19da8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34533 } health_report { overall_health: HEALTHY } } }
I20260923 20:54:13.002863 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:12.997601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.97.3:34533)
0923 20:54:12.997863 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:12.997871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:12.997898 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:54:12.998060 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:12.998064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:12.998142 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:12.998275 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:12.998681 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:12.999400 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.000611 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:13.001071 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.001204 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:13.001575 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:13.002259 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:13.002388 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:13.002473 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 20:54:13.002547 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:54:13.007063 7492 raft_consensus.cc:1261] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Refusing update from remote peer 8ee911db158c4ca6bf09fee02e19da8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:13.007325 7638 raft_consensus.cc:1261] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 1 FOLLOWER]: Refusing update from remote peer 8ee911db158c4ca6bf09fee02e19da8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:13.007877 7935 consensus_queue.cc:1035] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "16771be618554d49947f6fc449f5053b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:13.008229 7932 consensus_queue.cc:1035] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dece15634329499a8b2456f9ceae3f0d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:13.010519 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.009903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.1:34975)
0923 20:54:13.010233 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.010330 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
I20260923 20:54:13.011824 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.011005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.2:59393)
0923 20:54:13.011522 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.011640 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
I20260923 20:54:13.015987 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.015544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.1:36679)
0923 20:54:13.015746 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.015838 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 20:54:13.021344 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.018195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:44215)
0923 20:54:13.020862 (+ 2667us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.021100 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2503}
I20260923 20:54:13.028995 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.028070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.3:35967)
0923 20:54:13.028688 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.028798 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497}
I20260923 20:54:13.034127 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.033604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.97.2:33245)
0923 20:54:13.033792 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.033916 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 20:54:13.046685 7358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:13.046991 7358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:13.052886 7358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c52ce98709249a5912a4ef7e67c236c: Sending DeleteTablet for 3 replicas of tablet 7f483e6d93a448bcae05c8821ebe8e29
I20260923 20:54:13.053959 7472 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f483e6d93a448bcae05c8821ebe8e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:13 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33722
I20260923 20:54:13.054324 7759 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f483e6d93a448bcae05c8821ebe8e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:13 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:49764
I20260923 20:54:13.054396 7618 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f483e6d93a448bcae05c8821ebe8e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:13 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40270
I20260923 20:54:13.056025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 7346
I20260923 20:54:13.066439 7948 tablet_replica.cc:330] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: stopping tablet replica
I20260923 20:54:13.066409 7949 tablet_replica.cc:330] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: stopping tablet replica
I20260923 20:54:13.066675 7948 raft_consensus.cc:2229] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:13.066682 7949 raft_consensus.cc:2229] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:54:13.066999 7948 raft_consensus.cc:2258] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:13.068403 7948 ts_tablet_manager.cc:1905] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:13.070312 7949 raft_consensus.cc:2258] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:13.072531 7950 tablet_replica.cc:330] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: stopping tablet replica
I20260923 20:54:13.072865 7950 raft_consensus.cc:2229] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:13.072886 7949 ts_tablet_manager.cc:1905] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:13.073215 7950 raft_consensus.cc:2258] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:13.074487 7950 ts_tablet_manager.cc:1905] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:13.075393 7948 ts_tablet_manager.cc:1918] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:13.075536 7948 log.cc:1196] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/7f483e6d93a448bcae05c8821ebe8e29
I20260923 20:54:13.076088 7948 ts_tablet_manager.cc:1939] T 7f483e6d93a448bcae05c8821ebe8e29 P 16771be618554d49947f6fc449f5053b: Deleting consensus metadata
I20260923 20:54:13.077299 7344 catalog_manager.cc:4841] TS 16771be618554d49947f6fc449f5053b (127.0.97.1:41167): tablet 7f483e6d93a448bcae05c8821ebe8e29 (table test-table [id=0a8d7c8e174548388ece141804e9210f]) successfully deleted
I20260923 20:54:13.082052 7950 ts_tablet_manager.cc:1918] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:13.082300 7950 log.cc:1196] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/7f483e6d93a448bcae05c8821ebe8e29
I20260923 20:54:13.082828 7950 ts_tablet_manager.cc:1939] T 7f483e6d93a448bcae05c8821ebe8e29 P dece15634329499a8b2456f9ceae3f0d: Deleting consensus metadata
I20260923 20:54:13.084105 7345 catalog_manager.cc:4841] TS dece15634329499a8b2456f9ceae3f0d (127.0.97.2:38083): tablet 7f483e6d93a448bcae05c8821ebe8e29 (table test-table [id=0a8d7c8e174548388ece141804e9210f]) successfully deleted
I20260923 20:54:13.084239 7949 ts_tablet_manager.cc:1918] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:13.084431 7949 log.cc:1196] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/7f483e6d93a448bcae05c8821ebe8e29
I20260923 20:54:13.085003 7949 ts_tablet_manager.cc:1939] T 7f483e6d93a448bcae05c8821ebe8e29 P 8ee911db158c4ca6bf09fee02e19da8b: Deleting consensus metadata
I20260923 20:54:13.086138 7345 catalog_manager.cc:4841] TS 8ee911db158c4ca6bf09fee02e19da8b (127.0.97.3:34533): tablet 7f483e6d93a448bcae05c8821ebe8e29 (table test-table [id=0a8d7c8e174548388ece141804e9210f]) successfully deleted
I20260923 20:54:13.087383 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.057399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.97.1:41167)
0923 20:54:13.057638 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:13.057647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:13.068267 (+ 10620us) server_negotiation.cc:407] Connection header received
0923 20:54:13.068705 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:13.068714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:13.068828 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:13.068919 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:13.072157 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:13.073328 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.075256 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:13.075904 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.076032 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:13.079715 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:13.079734 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:13.079754 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:13.079816 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:13.083887 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:13.084875 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:13.084889 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:13.084896 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:13.085085 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:13.085560 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:13.085566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:13.085570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:13.086094 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:13.086476 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:13.086726 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 20:54:13.087000 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:54:13.088130 7532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:13.181355 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 7346
I20260923 20:54:13.202245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 7549
I20260923 20:54:13.217341 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.203137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.97.2:38083)
0923 20:54:13.203676 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:13.203682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:13.203703 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:54:13.203770 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:13.203774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:13.203831 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:13.203964 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:13.205546 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:13.206330 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.207644 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:13.208043 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.208181 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:13.211100 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:13.211118 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:13.211136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:13.211182 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:13.214518 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:13.215254 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:13.215261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:13.215263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:13.215388 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:13.215762 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:13.215769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:13.215772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:13.216128 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:13.216431 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:13.216749 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 20:54:13.216964 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260923 20:54:13.218206 7678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:13.341638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 7549
I20260923 20:54:13.364457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 7696
I20260923 20:54:13.381134 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.365291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.0.97.3:34533)
0923 20:54:13.365468 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:13.365473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:13.365496 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:54:13.365691 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:13.365694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:13.365754 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:13.365886 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:13.367771 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:13.368645 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.370514 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:13.370935 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:13.371055 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:13.374502 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:13.374516 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:13.374533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:13.374573 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:13.377775 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:13.378742 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:13.378749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:13.378752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:13.378916 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:13.379347 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:13.379350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:13.379352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:13.379953 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:13.380225 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:13.380569 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 20:54:13.380759 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:54:13.382097 7819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:13.428911 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.428164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45638)
0923 20:54:13.428647 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.428751 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":263,"threads_started":1}
I20260923 20:54:13.430334 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:13.429975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43651 (local address 127.0.0.1:45654)
0923 20:54:13.430126 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:13.430201 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:54:13.515733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 7696
I20260923 20:54:13.536921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 7032
I20260923 20:54:13.538081 7200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:13.653134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 7032
I20260923 20:54:13.674254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 7131
I20260923 20:54:13.675266 7389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:13.794234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 7131
2026-09-23T20:54:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9855 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:54:13 dist-test-slave-plcg krb5kdc[7961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:54:13 dist-test-slave-plcg krb5kdc[7961](info): set up 2 sockets
Sep 23 20:54:13 dist-test-slave-plcg krb5kdc[7961](info): commencing operation
krb5kdc: starting...
W20260923 20:54:14.914955 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:54:15 dist-test-slave-plcg krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260923 20:54:15.143265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38703
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:15.481734 7974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:15.482187 7974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:15.482347 7974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:15.490528 7974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:15.490687 7974 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:54:15.490778 7974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:15.490897 7974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:15.490981 7974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:15.491082 7974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:15.501513 7974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:15.502945 7974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:15.504395 7974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:15.517182 7979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:15.517227 7980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:15.517257 7982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:15.517899 7974 server_base.cc:1034] running on GCE node
I20260923 20:54:15.518774 7974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:15.520303 7974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:15.521585 7974 hybrid_clock.cc:648] HybridClock initialized: now 1790196855521496 us; error 66 us; skew 500 ppm
I20260923 20:54:15.522034 7974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:15 dist-test-slave-plcg krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:15.525609 7974 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260923 20:54:15.527397 7974 webserver.cc:460] Webserver started at http://127.0.97.62:45037/ using document root <none> and password file <none>
I20260923 20:54:15.528366 7974 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:15.528512 7974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:15.528890 7974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:15.532191 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "ec9cd7e252c143cd83a0b620c0c706a9"
format_stamp: "Formatted at 2026-09-23 20:54:15 on dist-test-slave-plcg"
server_key: "06701a4eae450e65977970eaf8882679"
server_key_iv: "8da112303a701bceb2e122deb1cd1bb4"
server_key_version: "encryptionkey@0"
I20260923 20:54:15.533111 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "ec9cd7e252c143cd83a0b620c0c706a9"
format_stamp: "Formatted at 2026-09-23 20:54:15 on dist-test-slave-plcg"
server_key: "06701a4eae450e65977970eaf8882679"
server_key_iv: "8da112303a701bceb2e122deb1cd1bb4"
server_key_version: "encryptionkey@0"
I20260923 20:54:15.539000 7974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:54:15.543130 7989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:15.544631 7974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:54:15.544878 7974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "ec9cd7e252c143cd83a0b620c0c706a9"
format_stamp: "Formatted at 2026-09-23 20:54:15 on dist-test-slave-plcg"
server_key: "06701a4eae450e65977970eaf8882679"
server_key_iv: "8da112303a701bceb2e122deb1cd1bb4"
server_key_version: "encryptionkey@0"
I20260923 20:54:15.545135 7974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:15.565543 7974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:15.570037 7974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:15.570336 7974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:15.593096 7974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38703
I20260923 20:54:15.593123 8050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38703 every 8 connection(s)
I20260923 20:54:15.594621 7974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:15.595602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 7974
I20260923 20:54:15.595837 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:54:15.596521 388 external_mini_cluster.cc:1428] Setting key 2c5a3064846f244fbd535ac0d2a20c53
I20260923 20:54:15.599244 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:15.610196 8051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Bootstrap starting.
Sep 23 20:54:15 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:15.613781 8051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:15.614976 8051 log.cc:824] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:15.618494 8051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: No bootstrap required, opened a new log
I20260923 20:54:15.619733 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:15.598562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.0.97.62:38703)
0923 20:54:15.599440 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:15.599452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:15.599492 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:54:15.601124 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:15.601145 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:15.601535 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:15.601969 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:15.603438 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:15.604806 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:15.606196 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:15.607054 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:15.607331 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:15.611638 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:15.611648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:15.611664 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:15.611710 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:15.614991 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:15.615982 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:15.615995 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:15.616006 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:15.616243 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:15.616749 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:15.616753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:15.616756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:15.617333 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:15.617565 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:15.618190 (+ 625us) server_negotiation.cc:299] Negotiation successful
0923 20:54:15.618693 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":353,"threads_started":1}
I20260923 20:54:15.626739 8051 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38703 } }
I20260923 20:54:15.627249 8051 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:15.627415 8051 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9cd7e252c143cd83a0b620c0c706a9, State: Initialized, Role: FOLLOWER
I20260923 20:54:15.628034 8051 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38703 } }
I20260923 20:54:15.628268 8051 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:15.628358 8051 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:15.628633 8051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:15.631536 8051 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38703 } }
I20260923 20:54:15.632102 8051 leader_election.cc:302] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec9cd7e252c143cd83a0b620c0c706a9; no voters:
I20260923 20:54:15.632782 8051 leader_election.cc:288] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:54:15.632979 8056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:15.633760 8056 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [term 1 LEADER]: Becoming Leader. State: Replica: ec9cd7e252c143cd83a0b620c0c706a9, State: Running, Role: LEADER
I20260923 20:54:15.634346 8056 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38703 } }
I20260923 20:54:15.635051 8051 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:15.638082 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9cd7e252c143cd83a0b620c0c706a9. Latest consensus state: current_term: 1 leader_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38703 } } }
I20260923 20:54:15.638872 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:15.637683 8057 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9cd7e252c143cd83a0b620c0c706a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38703 } } }
I20260923 20:54:15.643152 8057 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9 [sys.catalog]: This master's current role is: LEADER
W20260923 20:54:15.650605 8072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:54:15.650781 8072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:54:15.650933 8062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:15.652635 8062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:15.661100 8062 catalog_manager.cc:1347] Generated new cluster ID: 114e1b04186c4ee0a388d6501fe5cbe8
I20260923 20:54:15.661247 8062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:15.678172 8062 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:54:15.679515 8062 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:15.686448 8062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Generated new TSK 0
I20260923 20:54:15.687479 8062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260923 20:54:15.745272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38703
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:16.096463 8077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:16.096822 8077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:16.097121 8077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:16.105060 8077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:16.105254 8077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:16.105403 8077 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:54:16.105540 8077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:16.115651 8077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:16.117046 8077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:16.118472 8077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:16.132169 8083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:16.132186 8085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:16.132915 8082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:16.133009 8077 server_base.cc:1034] running on GCE node
I20260923 20:54:16.133692 8077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:16.134766 8077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:16.135993 8077 hybrid_clock.cc:648] HybridClock initialized: now 1790196856135936 us; error 36 us; skew 500 ppm
I20260923 20:54:16.136355 8077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:16 dist-test-slave-plcg krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:16.145546 8077 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 20:54:16.147336 8077 webserver.cc:460] Webserver started at http://127.0.97.1:35305/ using document root <none> and password file <none>
I20260923 20:54:16.148082 8077 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:16.148218 8077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:16.148609 8077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:16.151820 8077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "e9809bbf187142ed8c59f10c4b5415ff"
format_stamp: "Formatted at 2026-09-23 20:54:16 on dist-test-slave-plcg"
server_key: "0087d6082b2076bb5a328b2c9590c368"
server_key_iv: "b75a711ab69573d49ccb5e403a919409"
server_key_version: "encryptionkey@0"
I20260923 20:54:16.152655 8077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e9809bbf187142ed8c59f10c4b5415ff"
format_stamp: "Formatted at 2026-09-23 20:54:16 on dist-test-slave-plcg"
server_key: "0087d6082b2076bb5a328b2c9590c368"
server_key_iv: "b75a711ab69573d49ccb5e403a919409"
server_key_version: "encryptionkey@0"
I20260923 20:54:16.158275 8077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:54:16.166332 8092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:16.168793 8077 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260923 20:54:16.169032 8077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "e9809bbf187142ed8c59f10c4b5415ff"
format_stamp: "Formatted at 2026-09-23 20:54:16 on dist-test-slave-plcg"
server_key: "0087d6082b2076bb5a328b2c9590c368"
server_key_iv: "b75a711ab69573d49ccb5e403a919409"
server_key_version: "encryptionkey@0"
I20260923 20:54:16.169399 8077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:16.190654 8077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:16.196743 8077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:16.197072 8077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:16.200327 8077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:16.200457 8077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:16.200595 8077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:16.200701 8077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:54:16 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:16.234750 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.202085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56673 (local address 127.0.97.62:38703)
0923 20:54:16.202632 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:16.202639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:16.203215 (+ 576us) server_negotiation.cc:407] Connection header received
0923 20:54:16.206654 (+ 3439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:16.206660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:16.206767 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:16.206876 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:16.210429 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.211653 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.215583 (+ 3930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.216381 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.216527 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.223615 (+ 7088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:16.223625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:16.223633 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:16.223693 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:16.228164 (+ 4471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.229621 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.229631 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.229637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.229860 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.230608 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.230616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.230622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.231073 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:16.231261 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:16.234111 (+ 2850us) server_negotiation.cc:299] Negotiation successful
0923 20:54:16.234400 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
I20260923 20:54:16.235980 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.201906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38703 (local address 127.0.97.1:56673)
0923 20:54:16.203031 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:16.203082 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:16.206162 (+ 3080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:16.207564 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:16.207581 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:16.208779 (+ 1198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:16.210100 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.210135 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.211989 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.212001 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.215299 (+ 3298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.215328 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.217019 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.217027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.217518 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.218668 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:16.218710 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:16.223210 (+ 4500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:16.228450 (+ 5240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.228466 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.228498 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.229404 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.230030 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.230038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.230045 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.230419 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.231346 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:16.231358 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:16.232000 (+ 642us) client_negotiation.cc:769] Sending connection context
0923 20:54:16.232441 (+ 441us) client_negotiation.cc:240] Negotiation successful
0923 20:54:16.233192 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":530,"threads_started":1}
I20260923 20:54:16.259058 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.244369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58721 (local address 127.0.97.62:38703)
0923 20:54:16.244543 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:16.244550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:16.244666 (+ 116us) server_negotiation.cc:407] Connection header received
0923 20:54:16.244949 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:16.244954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:16.245039 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:16.245110 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:16.246318 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.247304 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.249973 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.250414 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.250583 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.252571 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:16.252576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:16.252580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:16.252613 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:16.255474 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.256465 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.256472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.256476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.256620 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.257180 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.257189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.257195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.257695 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:16.257957 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:16.258424 (+ 467us) server_negotiation.cc:299] Negotiation successful
0923 20:54:16.258674 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:54:16.259224 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.243815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38703 (local address 127.0.97.1:58721)
0923 20:54:16.244554 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:16.244575 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:16.244784 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:16.245268 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:16.245271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:16.245715 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:16.246163 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.246173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.247496 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.247504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.249704 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.249730 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.250732 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.250743 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.250900 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.251582 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:16.251601 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:16.252343 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:16.255639 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.255644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.255649 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.256326 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.256743 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.256748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.256752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.257018 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.257936 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:16.257948 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:16.258137 (+ 189us) client_negotiation.cc:769] Sending connection context
0923 20:54:16.258325 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 20:54:16.258713 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":403,"threads_started":1}
I20260923 20:54:16.268857 8077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38723
I20260923 20:54:16.268862 8212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38723 every 8 connection(s)
I20260923 20:54:16.270391 8077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:16.272432 8213 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38703
I20260923 20:54:16.272684 8213 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:16.273382 8213 heartbeater.cc:511] Master 127.0.97.62:38703 requested a full tablet report, sending...
I20260923 20:54:16.275619 8006 ts_manager.cc:194] Registered new tserver with Master: e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723)
I20260923 20:54:16.277096 8006 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56673
I20260923 20:54:16.280758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 8077
I20260923 20:54:16.280964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:54:16.281589 388 external_mini_cluster.cc:1428] Setting key 2aadfc22010a5c917018a106bfbae942
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260923 20:54:16.330513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38703
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:16.664175 8217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:16.664479 8217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:16.664577 8217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:16.672225 8217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:16.672354 8217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:16.672442 8217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:54:16.672523 8217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:16.682763 8217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:16.684039 8217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:16.685334 8217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:16.698762 8222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:16.698782 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:16.698704 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:16.699628 8217 server_base.cc:1034] running on GCE node
I20260923 20:54:16.700232 8217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:16.701269 8217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:16.702529 8217 hybrid_clock.cc:648] HybridClock initialized: now 1790196856702464 us; error 46 us; skew 500 ppm
I20260923 20:54:16.702852 8217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:16 dist-test-slave-plcg krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:16.706074 8217 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 20:54:16.707597 8217 webserver.cc:460] Webserver started at http://127.0.97.2:40863/ using document root <none> and password file <none>
I20260923 20:54:16.708287 8217 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:16.708385 8217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:16.708642 8217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:16.711686 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "383c8e7c35f9483c815ff6817bc21c49"
format_stamp: "Formatted at 2026-09-23 20:54:16 on dist-test-slave-plcg"
server_key: "48fcf0cdf6e370e7df18316f34cb2446"
server_key_iv: "95921bac4648a60b115348306552fcd7"
server_key_version: "encryptionkey@0"
I20260923 20:54:16.712500 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "383c8e7c35f9483c815ff6817bc21c49"
format_stamp: "Formatted at 2026-09-23 20:54:16 on dist-test-slave-plcg"
server_key: "48fcf0cdf6e370e7df18316f34cb2446"
server_key_iv: "95921bac4648a60b115348306552fcd7"
server_key_version: "encryptionkey@0"
I20260923 20:54:16.718096 8217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:54:16.722098 8232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:16.723604 8217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:54:16.723814 8217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "383c8e7c35f9483c815ff6817bc21c49"
format_stamp: "Formatted at 2026-09-23 20:54:16 on dist-test-slave-plcg"
server_key: "48fcf0cdf6e370e7df18316f34cb2446"
server_key_iv: "95921bac4648a60b115348306552fcd7"
server_key_version: "encryptionkey@0"
I20260923 20:54:16.724069 8217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:16.744645 8217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:16.749470 8217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:16.749761 8217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:16.752466 8217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:16.752584 8217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:16.752736 8217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:16.752820 8217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:54:16 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:16.781086 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.753288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43613 (local address 127.0.97.62:38703)
0923 20:54:16.753469 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:16.753475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:16.755018 (+ 1543us) server_negotiation.cc:407] Connection header received
0923 20:54:16.758204 (+ 3186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:16.758210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:16.758318 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:16.758453 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:16.761768 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.762765 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.766367 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.766912 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.767067 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.772530 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:16.772543 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:16.772547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:16.772610 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:16.776531 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.777974 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.777979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.777982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.778178 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.778911 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.778917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.778920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.779257 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:16.779430 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:16.780455 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0923 20:54:16.780701 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:54:16.782755 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.753928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38703 (local address 127.0.97.2:43613)
0923 20:54:16.754813 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:16.754866 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:16.757738 (+ 2872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:16.758831 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:16.758848 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:16.760378 (+ 1530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:16.761502 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.761544 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.763025 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.763038 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.766134 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.766162 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.767455 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.767467 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.767807 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.768721 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:16.768746 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:16.772258 (+ 3512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:16.776870 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.776887 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.776924 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.777732 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.778345 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.778356 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.778359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.778717 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.779423 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:16.779432 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:16.780019 (+ 587us) client_negotiation.cc:769] Sending connection context
0923 20:54:16.780346 (+ 327us) client_negotiation.cc:240] Negotiation successful
0923 20:54:16.780978 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":299,"threads_started":1}
I20260923 20:54:16.798751 8217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38341
I20260923 20:54:16.798815 8351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38341 every 8 connection(s)
I20260923 20:54:16.800209 8217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:16.802572 8352 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38703
I20260923 20:54:16.802742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 8217
I20260923 20:54:16.802861 8352 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:16.802907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:54:16.803365 388 external_mini_cluster.cc:1428] Setting key 62d6dae7dcc95acdf5321b451ee10e6c
I20260923 20:54:16.803716 8352 heartbeater.cc:511] Master 127.0.97.62:38703 requested a full tablet report, sending...
I20260923 20:54:16.805434 8005 ts_manager.cc:194] Registered new tserver with Master: 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341)
I20260923 20:54:16.806733 8005 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43613
I20260923 20:54:16.808781 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.790935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57697 (local address 127.0.97.62:38703)
0923 20:54:16.791220 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:16.791227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:16.791738 (+ 511us) server_negotiation.cc:407] Connection header received
0923 20:54:16.792217 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:16.792226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:16.792352 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:16.792482 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:16.793479 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.794993 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.797082 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.797893 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.798064 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.800143 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:16.800152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:16.800161 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:16.800216 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:16.803911 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.804981 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.804988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.804992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.805133 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:16.805608 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:16.805613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:16.805616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:16.807429 (+ 1813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:16.807590 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:16.808123 (+ 533us) server_negotiation.cc:299] Negotiation successful
0923 20:54:16.808373 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:54:16.808828 8336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:16.790508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38703 (local address 127.0.97.2:57697)
0923 20:54:16.791618 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:16.791635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:16.791834 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:16.792543 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:16.792548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:16.792996 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:16.793297 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.793306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.795190 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.795197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.796893 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:16.796907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:16.798173 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:16.798182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:16.798390 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:16.799095 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:16.799114 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:16.799912 (+ 798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:16.804091 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.804097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.804101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.804629 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.805282 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:16.805287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:16.805291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:16.805478 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:16.807592 (+ 2114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:16.807597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:16.807808 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 20:54:16.807999 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 20:54:16.808361 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":490,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 20:54:16.858244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38703
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:17.197402 8356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:17.197726 8356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:17.197886 8356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:17.205404 8356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:17.205530 8356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:17.205614 8356 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:54:17.205699 8356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:17.215865 8356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:17.217131 8356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:17.218554 8356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:17.232031 8361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:17.232153 8364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:17.232113 8362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:17.232965 8356 server_base.cc:1034] running on GCE node
I20260923 20:54:17.233682 8356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:17.234841 8356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:17.236137 8356 hybrid_clock.cc:648] HybridClock initialized: now 1790196857236046 us; error 51 us; skew 500 ppm
I20260923 20:54:17.236546 8356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:17 dist-test-slave-plcg krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:17.240216 8356 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 20:54:17.241921 8356 webserver.cc:460] Webserver started at http://127.0.97.3:37205/ using document root <none> and password file <none>
I20260923 20:54:17.242669 8356 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:17.242811 8356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:17.243139 8356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:17.246872 8356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "d06630c0491144b7ab6614f7adac8649"
format_stamp: "Formatted at 2026-09-23 20:54:17 on dist-test-slave-plcg"
server_key: "0c550b252bb8e580eff9762da59bf8d3"
server_key_iv: "320c41f7e2a1ed115ea0925021f3e78d"
server_key_version: "encryptionkey@0"
I20260923 20:54:17.248116 8356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d06630c0491144b7ab6614f7adac8649"
format_stamp: "Formatted at 2026-09-23 20:54:17 on dist-test-slave-plcg"
server_key: "0c550b252bb8e580eff9762da59bf8d3"
server_key_iv: "320c41f7e2a1ed115ea0925021f3e78d"
server_key_version: "encryptionkey@0"
I20260923 20:54:17.253799 8356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:54:17.257491 8371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:17.259033 8356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:54:17.259243 8356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "d06630c0491144b7ab6614f7adac8649"
format_stamp: "Formatted at 2026-09-23 20:54:17 on dist-test-slave-plcg"
server_key: "0c550b252bb8e580eff9762da59bf8d3"
server_key_iv: "320c41f7e2a1ed115ea0925021f3e78d"
server_key_version: "encryptionkey@0"
I20260923 20:54:17.259516 8356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:17.280079 8213 heartbeater.cc:503] Master 127.0.97.62:38703 was elected leader, sending a full tablet report...
I20260923 20:54:17.281410 8356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:17.286365 8356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:17.286689 8356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:17.289757 8356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:17.289911 8356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 20:54:17.290097 8356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:17.290182 8356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:54:17 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:17.315966 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.290224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36211 (local address 127.0.97.62:38703)
0923 20:54:17.290437 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.290444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.291974 (+ 1530us) server_negotiation.cc:407] Connection header received
0923 20:54:17.294276 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.294286 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.294413 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.294574 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:17.297016 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.298528 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.301579 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.302445 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.302622 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.308491 (+ 5869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:17.308499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:17.308506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:17.308562 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:17.312209 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:17.313067 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:17.313076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:17.313082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:17.313280 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:17.313788 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:17.313791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:17.313793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:17.314181 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:17.314353 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.315344 (+ 991us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.315591 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:54:17.317018 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.290876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38703 (local address 127.0.97.3:36211)
0923 20:54:17.291765 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:17.291816 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:17.293903 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:17.294890 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:17.294905 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:17.295801 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:17.296746 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.296781 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.298752 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.298760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.301381 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.301397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.302718 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.302724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.302946 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.303777 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:17.303828 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:17.307891 (+ 4063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:17.312387 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:17.312403 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:17.312420 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:17.312904 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:17.313443 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:17.313448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:17.313454 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:17.313657 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:17.314346 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:17.314354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:17.314884 (+ 530us) client_negotiation.cc:769] Sending connection context
0923 20:54:17.315258 (+ 374us) client_negotiation.cc:240] Negotiation successful
0923 20:54:17.315824 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":374,"threads_started":1}
I20260923 20:54:17.339625 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.323169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51647 (local address 127.0.97.62:38703)
0923 20:54:17.323394 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.323400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.324186 (+ 786us) server_negotiation.cc:407] Connection header received
0923 20:54:17.324411 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.324417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.324505 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.324625 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:17.326057 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.327174 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.329243 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.329700 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.329926 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.333037 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:17.333041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:17.333045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:17.333079 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:17.336868 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:17.337589 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:17.337595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:17.337599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:17.337729 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:17.338132 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:17.338139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:17.338142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:17.338430 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:17.338588 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.339023 (+ 435us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.339255 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:54:17.339717 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.323250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38703 (local address 127.0.97.3:51647)
0923 20:54:17.324052 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:17.324086 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:17.324270 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:17.324845 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:17.324850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:17.325401 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:17.325884 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.325899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.327343 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.327350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.329070 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.329088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.330043 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.330051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.330265 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.331299 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:17.331344 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:17.332797 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:17.337024 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:17.337030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:17.337034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:17.337456 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:17.337875 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:17.337882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:17.337886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:17.338020 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:17.338549 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:17.338553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:17.338752 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 20:54:17.338899 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:54:17.339246 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":474,"threads_started":1}
I20260923 20:54:17.349968 8356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36875
I20260923 20:54:17.350008 8490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36875 every 8 connection(s)
I20260923 20:54:17.351300 8356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:17.351686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 8356
I20260923 20:54:17.351886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:54:17.352535 388 external_mini_cluster.cc:1428] Setting key 267f210f0192cfaac5d35c078fb1d2f9
I20260923 20:54:17.353603 8491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38703
I20260923 20:54:17.353901 8491 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:17.354638 8491 heartbeater.cc:511] Master 127.0.97.62:38703 requested a full tablet report, sending...
I20260923 20:54:17.355916 8006 ts_manager.cc:194] Registered new tserver with Master: d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:17.356727 8006 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36211
I20260923 20:54:17.368069 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:54:17 dist-test-slave-plcg krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:54:17 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:17 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:54:17.419445 8101 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:17.408166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.97.62:38703)
0923 20:54:17.408414 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.408421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.408613 (+ 192us) server_negotiation.cc:407] Connection header received
0923 20:54:17.408759 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.408763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.408827 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.408924 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:17.410361 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.411093 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.412367 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.412819 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.412932 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.419165 (+ 6233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41992: BlockingRecv error: recv got EOF from 127.0.0.1:41992 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 23 20:54:17 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196857, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:17.436771 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.422685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.0.97.62:38703)
0923 20:54:17.422883 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.422888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.423216 (+ 328us) server_negotiation.cc:407] Connection header received
0923 20:54:17.423363 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.423367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.423449 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.423547 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:17.424717 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.425493 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.427045 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.427559 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.427715 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.431526 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:17.431533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:17.431536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:17.431575 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:17.434330 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:17.434992 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:17.434997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:17.434999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:17.435117 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:17.435550 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:17.435554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:17.435556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:17.435851 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:17.436107 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.436321 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.436503 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:54:17.440804 8006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:54:17.447865 8006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:17.485687 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.461864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.97.3:36875)
0923 20:54:17.469434 (+ 7570us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.469448 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.477107 (+ 7659us) server_negotiation.cc:407] Connection header received
0923 20:54:17.477373 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.477382 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.477667 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.477879 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:17.479024 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.480198 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.482851 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.484050 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.484184 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.484451 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.485094 (+ 643us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.485341 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7303,"thread_start_us":7079,"threads_started":1}
I20260923 20:54:17.485695 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.460983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36875 (local address 127.0.0.1:54732)
0923 20:54:17.476968 (+ 15985us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:17.476986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:17.477140 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:17.478154 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:17.478164 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:17.478209 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:17.478748 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.478764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.480374 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.480379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.482669 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.482683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.484627 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.484633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.484828 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.484883 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 20:54:17.485019 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 20:54:17.485231 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15893,"mutex_wait_us":6,"spinlock_wait_cycles":4480,"thread_start_us":15126,"threads_started":1}
I20260923 20:54:17.487263 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.460251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38341 (local address 127.0.0.1:57410)
0923 20:54:17.475362 (+ 15111us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:17.475441 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:17.475819 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:17.478155 (+ 2336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:17.478163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:17.478228 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:17.478826 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.478837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.480734 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.480739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.483504 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.483519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.485323 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.485331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.485513 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.485549 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:54:17.485708 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 20:54:17.485905 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14652,"mutex_wait_us":33,"spinlock_wait_cycles":3840,"thread_start_us":16503,"threads_started":1}
I20260923 20:54:17.488088 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.460408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38723 (local address 127.0.0.1:40630)
0923 20:54:17.476332 (+ 15924us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:17.476360 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:17.477555 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:17.478461 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:17.478469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:17.478489 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:17.479073 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.479084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.480911 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.480916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.483505 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.483519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.487266 (+ 3747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.487273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.487467 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.487505 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:54:17.487597 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:54:17.487729 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15745,"mutex_wait_us":106,"thread_start_us":15547,"threads_started":1}
I20260923 20:54:17.488400 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.460981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.0.97.1:38723)
0923 20:54:17.471754 (+ 10773us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.471767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.476496 (+ 4729us) server_negotiation.cc:407] Connection header received
0923 20:54:17.477772 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.477780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.478040 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.478224 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:17.479263 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.480745 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.485480 (+ 4735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.487028 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.487200 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.487410 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.487722 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.487913 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10525,"thread_start_us":10336,"threads_started":1}
I20260923 20:54:17.488507 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.460324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.0.97.2:38341)
0923 20:54:17.477539 (+ 17215us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.477548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.477573 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:54:17.477658 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.477665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.477919 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.478420 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:17.479048 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.480566 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.483692 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.485066 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.485192 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.485377 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.487625 (+ 2248us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.487816 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17063,"thread_start_us":16914,"threads_started":1}
I20260923 20:54:17.489993 8421 tablet_service.cc:1467] Processing CreateTablet for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a (DEFAULT_TABLE table=test-table [id=0b9311ff86f34b38a1f99ab822d64877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:17.491546 8421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:17.491683 8282 tablet_service.cc:1467] Processing CreateTablet for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a (DEFAULT_TABLE table=test-table [id=0b9311ff86f34b38a1f99ab822d64877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:17.492161 8143 tablet_service.cc:1467] Processing CreateTablet for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a (DEFAULT_TABLE table=test-table [id=0b9311ff86f34b38a1f99ab822d64877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:17.492852 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:17.493901 8143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:17.504384 8515 tablet_bootstrap.cc:492] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Bootstrap starting.
I20260923 20:54:17.506409 8516 tablet_bootstrap.cc:492] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Bootstrap starting.
I20260923 20:54:17.508921 8515 tablet_bootstrap.cc:654] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:17.508970 8517 tablet_bootstrap.cc:492] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Bootstrap starting.
I20260923 20:54:17.509996 8516 tablet_bootstrap.cc:654] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:17.510864 8515 log.cc:824] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:17.511186 8516 log.cc:824] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:17.512131 8517 tablet_bootstrap.cc:654] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:17.513280 8517 log.cc:824] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:17.515312 8516 tablet_bootstrap.cc:492] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: No bootstrap required, opened a new log
I20260923 20:54:17.515758 8516 ts_tablet_manager.cc:1397] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260923 20:54:17.515884 8517 tablet_bootstrap.cc:492] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: No bootstrap required, opened a new log
I20260923 20:54:17.516147 8517 ts_tablet_manager.cc:1397] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 20:54:17.521040 8515 tablet_bootstrap.cc:492] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: No bootstrap required, opened a new log
I20260923 20:54:17.521366 8515 ts_tablet_manager.cc:1397] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260923 20:54:17.524549 8517 raft_consensus.cc:348] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.524979 8517 raft_consensus.cc:374] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:17.525072 8517 raft_consensus.cc:729] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9809bbf187142ed8c59f10c4b5415ff, State: Initialized, Role: FOLLOWER
I20260923 20:54:17.525739 8517 consensus_queue.cc:260] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.526961 8516 raft_consensus.cc:348] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.527434 8516 raft_consensus.cc:374] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:17.527527 8516 raft_consensus.cc:729] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06630c0491144b7ab6614f7adac8649, State: Initialized, Role: FOLLOWER
I20260923 20:54:17.527683 8517 ts_tablet_manager.cc:1428] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20260923 20:54:17.528301 8516 consensus_queue.cc:260] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.529420 8491 heartbeater.cc:503] Master 127.0.97.62:38703 was elected leader, sending a full tablet report...
I20260923 20:54:17.530560 8516 ts_tablet_manager.cc:1428] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260923 20:54:17.535568 8515 raft_consensus.cc:348] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.535960 8515 raft_consensus.cc:374] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:17.536041 8515 raft_consensus.cc:729] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Initialized, Role: FOLLOWER
I20260923 20:54:17.536550 8515 consensus_queue.cc:260] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.537595 8352 heartbeater.cc:503] Master 127.0.97.62:38703 was elected leader, sending a full tablet report...
I20260923 20:54:17.538229 8515 ts_tablet_manager.cc:1428] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260923 20:54:17.552551 8353 tablet.cc:2369] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:54:17.603525 8492 tablet.cc:2369] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:54:17.758642 8521 raft_consensus.cc:482] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:17.758962 8521 raft_consensus.cc:504] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.760613 8521 leader_election.cc:288] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:17.770107 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.761167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42389 (local address 127.0.97.2:38341)
0923 20:54:17.761420 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.761428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.761964 (+ 536us) server_negotiation.cc:407] Connection header received
0923 20:54:17.762175 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.762181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.762261 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.762386 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:17.763048 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.764221 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.767564 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.768840 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.768994 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.769110 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.769670 (+ 560us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.769806 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:54:17.770102 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.761135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60013 (local address 127.0.97.3:36875)
0923 20:54:17.761344 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.761353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.761964 (+ 611us) server_negotiation.cc:407] Connection header received
0923 20:54:17.762176 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.762181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.762261 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.762366 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:17.763184 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.764221 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.767564 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.768840 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.768994 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.769110 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.769670 (+ 560us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.769808 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:54:17.770337 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.761006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38341 (local address 127.0.97.1:42389)
0923 20:54:17.761784 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:17.761809 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:17.762040 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:17.762397 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:17.762401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:17.762427 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:17.762883 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.762898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.764382 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.764387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.767304 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.767323 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.769166 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.769171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.769414 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.769515 (+ 101us) client_negotiation.cc:769] Sending connection context
0923 20:54:17.769621 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:54:17.769810 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":310,"spinlock_wait_cycles":25856,"thread_start_us":430,"threads_started":1}
I20260923 20:54:17.770557 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.761087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36875 (local address 127.0.97.1:60013)
0923 20:54:17.761835 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:17.761850 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:17.762000 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:17.762396 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:17.762401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:17.762470 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:17.763017 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.763029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.764385 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.764390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.767304 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:17.767323 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.769154 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.769161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:17.769411 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.769520 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 20:54:17.769621 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:54:17.769810 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":307,"spinlock_wait_cycles":57088,"thread_start_us":464,"threads_started":1}
I20260923 20:54:17.771440 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e9f4e03c5a43edb00b7bf5cf7fbe1a" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649" is_pre_election: true
I20260923 20:54:17.771446 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e9f4e03c5a43edb00b7bf5cf7fbe1a" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383c8e7c35f9483c815ff6817bc21c49" is_pre_election: true
I20260923 20:54:17.771917 8446 raft_consensus.cc:2454] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 0.
I20260923 20:54:17.771917 8302 raft_consensus.cc:2454] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 0.
I20260923 20:54:17.772609 8094 leader_election.cc:302] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, d06630c0491144b7ab6614f7adac8649, e9809bbf187142ed8c59f10c4b5415ff; no voters:
W20260923 20:54:17.772993 8214 tablet.cc:2369] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:54:17.773514 8521 raft_consensus.cc:2781] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:17.773685 8521 raft_consensus.cc:482] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:17.773846 8521 raft_consensus.cc:3037] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:17.776356 8521 raft_consensus.cc:504] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.777022 8521 leader_election.cc:288] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 election: Requested vote from peers 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:17.777514 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e9f4e03c5a43edb00b7bf5cf7fbe1a" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383c8e7c35f9483c815ff6817bc21c49"
I20260923 20:54:17.777540 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e9f4e03c5a43edb00b7bf5cf7fbe1a" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649"
I20260923 20:54:17.777731 8302 raft_consensus.cc:3037] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:17.777751 8446 raft_consensus.cc:3037] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:17.779955 8446 raft_consensus.cc:2454] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 1.
I20260923 20:54:17.779971 8302 raft_consensus.cc:2454] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 1.
I20260923 20:54:17.780449 8094 leader_election.cc:302] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d06630c0491144b7ab6614f7adac8649, e9809bbf187142ed8c59f10c4b5415ff; no voters:
I20260923 20:54:17.780853 8521 raft_consensus.cc:2781] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:17.781281 8521 raft_consensus.cc:686] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 1 LEADER]: Becoming Leader. State: Replica: e9809bbf187142ed8c59f10c4b5415ff, State: Running, Role: LEADER
I20260923 20:54:17.781761 8521 consensus_queue.cc:237] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.786669 8006 catalog_manager.cc:5495] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff reported cstate change: term changed from 0 to 1, leader changed from <none> to e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e9809bbf187142ed8c59f10c4b5415ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } health_report { overall_health: UNKNOWN } } }
I20260923 20:54:17.857643 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:17.852158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.97.1:38723)
0923 20:54:17.852398 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:17.852406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:17.852496 (+ 90us) server_negotiation.cc:407] Connection header received
0923 20:54:17.852770 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:17.852774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:17.852867 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:17.853031 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:17.853400 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.854162 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.855500 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:17.855905 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:17.856036 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:17.856480 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:17.857034 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:17.857193 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:17.857274 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 20:54:17.857357 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:54:17.866283 8446 raft_consensus.cc:1261] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Refusing update from remote peer e9809bbf187142ed8c59f10c4b5415ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:17.866283 8302 raft_consensus.cc:1261] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Refusing update from remote peer e9809bbf187142ed8c59f10c4b5415ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:17.867331 8521 consensus_queue.cc:1035] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:17.867648 8526 consensus_queue.cc:1035] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:17.899454 8006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:17.899842 8006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:17.904749 8006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Sending DeleteTablet for 3 replicas of tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a
I20260923 20:54:17.906087 8421 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54732
I20260923 20:54:17.906097 8282 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57410
I20260923 20:54:17.906126 8143 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:54:17.909651 8006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:54:17.911623 8006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:17.922322 8142 tablet_service.cc:1467] Processing CreateTablet for tablet e4834d774fc84caf93a9884fe468d5d7 (DEFAULT_TABLE table=test-table [id=f3faeb8d908849e7884fd18f1dac9c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:17.922976 8142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4834d774fc84caf93a9884fe468d5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:17.924906 8420 tablet_service.cc:1467] Processing CreateTablet for tablet e4834d774fc84caf93a9884fe468d5d7 (DEFAULT_TABLE table=test-table [id=f3faeb8d908849e7884fd18f1dac9c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:17.925495 8420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4834d774fc84caf93a9884fe468d5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:17.926986 8281 tablet_service.cc:1467] Processing CreateTablet for tablet e4834d774fc84caf93a9884fe468d5d7 (DEFAULT_TABLE table=test-table [id=f3faeb8d908849e7884fd18f1dac9c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:17.927604 8281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4834d774fc84caf93a9884fe468d5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:17.937873 8517 tablet_bootstrap.cc:492] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: Bootstrap starting.
I20260923 20:54:17.940447 8517 tablet_bootstrap.cc:654] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:17.941192 8515 tablet_bootstrap.cc:492] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: Bootstrap starting.
I20260923 20:54:17.943008 8515 tablet_bootstrap.cc:654] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:17.946090 8515 tablet_bootstrap.cc:492] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: No bootstrap required, opened a new log
I20260923 20:54:17.946327 8515 ts_tablet_manager.cc:1397] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 20:54:17.947214 8515 raft_consensus.cc:348] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.947621 8515 raft_consensus.cc:374] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:17.947789 8515 raft_consensus.cc:729] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Initialized, Role: FOLLOWER
I20260923 20:54:17.948138 8515 consensus_queue.cc:260] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.949188 8516 tablet_bootstrap.cc:492] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: Bootstrap starting.
I20260923 20:54:17.949687 8515 ts_tablet_manager.cc:1428] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:54:17.950935 8516 tablet_bootstrap.cc:654] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:17.951865 8517 tablet_bootstrap.cc:492] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: No bootstrap required, opened a new log
I20260923 20:54:17.952028 8517 ts_tablet_manager.cc:1397] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20260923 20:54:17.952919 8517 raft_consensus.cc:348] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.953145 8517 raft_consensus.cc:374] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:17.953248 8517 raft_consensus.cc:729] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9809bbf187142ed8c59f10c4b5415ff, State: Initialized, Role: FOLLOWER
I20260923 20:54:17.953554 8517 consensus_queue.cc:260] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.955430 8517 ts_tablet_manager.cc:1428] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:17.963303 8516 tablet_bootstrap.cc:492] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: No bootstrap required, opened a new log
I20260923 20:54:17.963518 8516 ts_tablet_manager.cc:1397] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20260923 20:54:17.964661 8516 raft_consensus.cc:348] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.964984 8516 raft_consensus.cc:374] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:17.965075 8516 raft_consensus.cc:729] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06630c0491144b7ab6614f7adac8649, State: Initialized, Role: FOLLOWER
I20260923 20:54:17.965387 8516 consensus_queue.cc:260] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:17.969205 8516 ts_tablet_manager.cc:1428] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260923 20:54:17.976221 8540 tablet_replica.cc:330] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: stopping tablet replica
I20260923 20:54:17.976866 8540 raft_consensus.cc:2229] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:17.977500 8540 raft_consensus.cc:2258] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:17.978447 8541 tablet_replica.cc:330] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: stopping tablet replica
I20260923 20:54:17.979000 8541 raft_consensus.cc:2229] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:54:17.979957 8541 raft_consensus.cc:2258] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:17.980826 8539 tablet_replica.cc:330] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: stopping tablet replica
I20260923 20:54:17.981235 8539 raft_consensus.cc:2229] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:17.982177 8539 raft_consensus.cc:2258] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:17.983394 8540 ts_tablet_manager.cc:1905] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:17.983827 8539 ts_tablet_manager.cc:1905] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:17.988857 8541 ts_tablet_manager.cc:1905] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:17.989509 8539 ts_tablet_manager.cc:1918] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:17.989712 8539 log.cc:1196] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/a0e9f4e03c5a43edb00b7bf5cf7fbe1a
I20260923 20:54:17.990176 8539 ts_tablet_manager.cc:1939] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P 383c8e7c35f9483c815ff6817bc21c49: Deleting consensus metadata
I20260923 20:54:17.991721 7993 catalog_manager.cc:4841] TS 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341): tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a (table test-table [id=0b9311ff86f34b38a1f99ab822d64877]) successfully deleted
I20260923 20:54:17.995579 8540 ts_tablet_manager.cc:1918] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:17.995776 8540 log.cc:1196] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/a0e9f4e03c5a43edb00b7bf5cf7fbe1a
I20260923 20:54:17.996220 8540 ts_tablet_manager.cc:1939] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P d06630c0491144b7ab6614f7adac8649: Deleting consensus metadata
I20260923 20:54:17.996511 8541 ts_tablet_manager.cc:1918] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:17.996693 8541 log.cc:1196] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/a0e9f4e03c5a43edb00b7bf5cf7fbe1a
I20260923 20:54:17.997210 8541 ts_tablet_manager.cc:1939] T a0e9f4e03c5a43edb00b7bf5cf7fbe1a P e9809bbf187142ed8c59f10c4b5415ff: Deleting consensus metadata
I20260923 20:54:17.998432 7991 catalog_manager.cc:4841] TS d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875): tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a (table test-table [id=0b9311ff86f34b38a1f99ab822d64877]) successfully deleted
I20260923 20:54:17.998911 7992 catalog_manager.cc:4841] TS e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723): tablet a0e9f4e03c5a43edb00b7bf5cf7fbe1a (table test-table [id=0b9311ff86f34b38a1f99ab822d64877]) successfully deleted
I20260923 20:54:18.139989 8523 raft_consensus.cc:482] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:18.140228 8523 raft_consensus.cc:504] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.141801 8523 leader_election.cc:288] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:18.150959 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.142545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50057 (local address 127.0.97.3:36875)
0923 20:54:18.142806 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.142815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.143208 (+ 393us) server_negotiation.cc:407] Connection header received
0923 20:54:18.143513 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.143518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.143628 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.143742 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:18.144900 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.146121 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.148587 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.149520 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.149769 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.149894 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.150554 (+ 660us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.150696 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260923 20:54:18.151125 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.142131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48655 (local address 127.0.97.1:38723)
0923 20:54:18.142343 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.142352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.143601 (+ 1249us) server_negotiation.cc:407] Connection header received
0923 20:54:18.143775 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.143779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.143879 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.143973 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:18.144797 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.145637 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.148332 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.149660 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.149846 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.149974 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.150626 (+ 652us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.150800 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:54:18.151780 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.142177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36875 (local address 127.0.97.2:50057)
0923 20:54:18.143034 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:18.143058 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:18.143333 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:18.144189 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:18.144205 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:18.144246 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:18.144673 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.144683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.146274 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.146280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.148375 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.148388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.150093 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.150101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.150309 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.150384 (+ 75us) client_negotiation.cc:769] Sending connection context
0923 20:54:18.151243 (+ 859us) client_negotiation.cc:240] Negotiation successful
0923 20:54:18.151422 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":8,"spinlock_wait_cycles":1792,"thread_start_us":520,"threads_started":1}
I20260923 20:54:18.151897 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.142460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38723 (local address 127.0.97.2:48655)
0923 20:54:18.143474 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:18.143494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:18.143634 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:18.144031 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:18.144036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:18.144064 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:18.144584 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.144605 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.145786 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.145793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.148131 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.148148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.150141 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.150148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.150427 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.150465 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:54:18.151311 (+ 846us) client_negotiation.cc:240] Negotiation successful
0923 20:54:18.151463 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":181,"spinlock_wait_cycles":28288,"thread_start_us":647,"threads_started":1}
I20260923 20:54:18.152679 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4834d774fc84caf93a9884fe468d5d7" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649" is_pre_election: true
I20260923 20:54:18.152967 8446 raft_consensus.cc:2454] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 0.
I20260923 20:54:18.152797 8163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4834d774fc84caf93a9884fe468d5d7" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9809bbf187142ed8c59f10c4b5415ff" is_pre_election: true
I20260923 20:54:18.153337 8163 raft_consensus.cc:2454] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 0.
I20260923 20:54:18.153496 8234 leader_election.cc:302] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, d06630c0491144b7ab6614f7adac8649; no voters:
I20260923 20:54:18.153999 8523 raft_consensus.cc:2781] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:18.154148 8523 raft_consensus.cc:482] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:18.154255 8523 raft_consensus.cc:3037] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.157465 8523 raft_consensus.cc:504] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.158272 8523 leader_election.cc:288] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 election: Requested vote from peers e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:18.158859 8163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4834d774fc84caf93a9884fe468d5d7" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9809bbf187142ed8c59f10c4b5415ff"
I20260923 20:54:18.158888 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4834d774fc84caf93a9884fe468d5d7" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649"
I20260923 20:54:18.159101 8163 raft_consensus.cc:3037] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.159122 8446 raft_consensus.cc:3037] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.161542 8163 raft_consensus.cc:2454] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 1.
I20260923 20:54:18.162108 8235 leader_election.cc:302] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, e9809bbf187142ed8c59f10c4b5415ff; no voters:
I20260923 20:54:18.162210 8446 raft_consensus.cc:2454] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 1.
I20260923 20:54:18.162492 8523 raft_consensus.cc:2781] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:18.163053 8523 raft_consensus.cc:686] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 LEADER]: Becoming Leader. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Running, Role: LEADER
I20260923 20:54:18.163476 8523 consensus_queue.cc:237] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.167207 8006 catalog_manager.cc:5495] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:54:18 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:18.241529 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.224327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.97.62:38703)
0923 20:54:18.225401 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.225406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.225431 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:54:18.225489 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.225493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.225543 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.225657 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:18.226877 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.227775 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.229774 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.230272 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.230388 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.234881 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:18.234886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:18.234889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:18.234924 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:18.238027 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:18.239084 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:18.239089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:18.239091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:18.239272 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:18.239748 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:18.239751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:18.239753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:18.240098 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:18.240233 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.240797 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.241228 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":950,"thread_start_us":796,"threads_started":1}
I20260923 20:54:18.241557 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.224261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38703 (local address 127.0.0.1:41534)
0923 20:54:18.225130 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:18.225158 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:18.225340 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:18.225693 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:18.225700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:18.226505 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:18.226712 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.226723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.228011 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.228017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.229569 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.229587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.230482 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.230491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.230679 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.231496 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:18.231522 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:18.234632 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:18.238259 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:18.238279 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:18.238291 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:18.238923 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:18.239401 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:18.239408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:18.239413 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:18.239630 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:18.240245 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:18.240255 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:18.240489 (+ 234us) client_negotiation.cc:769] Sending connection context
0923 20:54:18.240692 (+ 203us) client_negotiation.cc:240] Negotiation successful
0923 20:54:18.241016 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":486,"threads_started":1}
I20260923 20:54:18.247686 8006 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41534:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:54:18.249064 8006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:18.257659 8282 tablet_service.cc:1467] Processing CreateTablet for tablet 76a2e3f0d39b4cd8b65ed080c6a35e91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6eaff031823d44658bb15bdebc43a900]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:54:18.257778 8143 tablet_service.cc:1467] Processing CreateTablet for tablet 76a2e3f0d39b4cd8b65ed080c6a35e91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6eaff031823d44658bb15bdebc43a900]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:54:18.258348 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a2e3f0d39b4cd8b65ed080c6a35e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.258558 8143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a2e3f0d39b4cd8b65ed080c6a35e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.262715 8421 tablet_service.cc:1467] Processing CreateTablet for tablet 76a2e3f0d39b4cd8b65ed080c6a35e91 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6eaff031823d44658bb15bdebc43a900]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:54:18.263337 8421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a2e3f0d39b4cd8b65ed080c6a35e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.265777 8517 tablet_bootstrap.cc:492] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: Bootstrap starting.
I20260923 20:54:18.267685 8517 tablet_bootstrap.cc:654] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.268159 8515 tablet_bootstrap.cc:492] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: Bootstrap starting.
I20260923 20:54:18.270085 8515 tablet_bootstrap.cc:654] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.273392 8516 tablet_bootstrap.cc:492] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: Bootstrap starting.
I20260923 20:54:18.275616 8516 tablet_bootstrap.cc:654] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.281313 8517 tablet_bootstrap.cc:492] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: No bootstrap required, opened a new log
I20260923 20:54:18.281656 8517 ts_tablet_manager.cc:1397] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260923 20:54:18.282829 8517 raft_consensus.cc:348] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.283357 8517 raft_consensus.cc:374] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.283515 8517 raft_consensus.cc:729] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9809bbf187142ed8c59f10c4b5415ff, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.283913 8517 consensus_queue.cc:260] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.284246 8515 tablet_bootstrap.cc:492] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: No bootstrap required, opened a new log
I20260923 20:54:18.284444 8515 ts_tablet_manager.cc:1397] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20260923 20:54:18.284673 8534 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.285001 8534 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:54:18.285292 8515 raft_consensus.cc:348] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.285557 8515 raft_consensus.cc:374] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.285651 8515 raft_consensus.cc:729] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.285944 8517 ts_tablet_manager.cc:1428] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:54:18.285969 8515 consensus_queue.cc:260] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.287145 8515 ts_tablet_manager.cc:1428] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:54:18.287328 8523 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.287721 8523 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:54:18.289191 8516 tablet_bootstrap.cc:492] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: No bootstrap required, opened a new log
I20260923 20:54:18.289439 8516 ts_tablet_manager.cc:1397] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260923 20:54:18.290387 8516 raft_consensus.cc:348] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.290717 8516 raft_consensus.cc:374] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.290849 8516 raft_consensus.cc:729] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06630c0491144b7ab6614f7adac8649, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.291170 8516 consensus_queue.cc:260] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.291785 8516 ts_tablet_manager.cc:1428] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:54:18.292232 8522 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.292639 8522 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:54:18.371454 8523 raft_consensus.cc:482] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:18.371690 8523 raft_consensus.cc:504] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.372431 8523 leader_election.cc:288] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:18.373117 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a2e3f0d39b4cd8b65ed080c6a35e91" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649" is_pre_election: true
I20260923 20:54:18.373119 8163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a2e3f0d39b4cd8b65ed080c6a35e91" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9809bbf187142ed8c59f10c4b5415ff" is_pre_election: true
I20260923 20:54:18.373402 8163 raft_consensus.cc:2454] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 0.
I20260923 20:54:18.373401 8446 raft_consensus.cc:2454] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 0.
I20260923 20:54:18.373937 8234 leader_election.cc:302] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, d06630c0491144b7ab6614f7adac8649; no voters:
I20260923 20:54:18.374284 8523 raft_consensus.cc:2781] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:18.374382 8523 raft_consensus.cc:482] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:18.374536 8523 raft_consensus.cc:3037] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.376992 8523 raft_consensus.cc:504] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.378074 8163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a2e3f0d39b4cd8b65ed080c6a35e91" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9809bbf187142ed8c59f10c4b5415ff"
I20260923 20:54:18.378345 8163 raft_consensus.cc:3037] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.378605 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a2e3f0d39b4cd8b65ed080c6a35e91" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649"
I20260923 20:54:18.378839 8446 raft_consensus.cc:3037] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.381439 8163 raft_consensus.cc:2454] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 1.
I20260923 20:54:18.381974 8235 leader_election.cc:302] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, e9809bbf187142ed8c59f10c4b5415ff; no voters:
I20260923 20:54:18.382299 8547 raft_consensus.cc:2781] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:18.382431 8446 raft_consensus.cc:2454] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 1.
I20260923 20:54:18.383104 8523 leader_election.cc:288] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 election: Requested vote from peers e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:18.384177 8547 raft_consensus.cc:686] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 LEADER]: Becoming Leader. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Running, Role: LEADER
I20260923 20:54:18.384543 8547 consensus_queue.cc:237] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.386536 8523 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: TxnStatusTablet state changed. Reason: New leader 383c8e7c35f9483c815ff6817bc21c49. Latest consensus state: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.386834 8523 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:54:18.387462 8561 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:54:18.387693 8561 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:54:18.388142 8006 catalog_manager.cc:5495] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } health_report { overall_health: UNKNOWN } } }
I20260923 20:54:18.416378 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.410073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.0.97.2:38341)
0923 20:54:18.410749 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.410753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.410772 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:54:18.410846 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.410850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.410927 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.411128 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:18.411546 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.412426 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.413767 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.414298 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.414445 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.414972 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:18.415589 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:18.415761 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.415895 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.415990 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":382,"threads_started":1}
I20260923 20:54:18.416345 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.410071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38341 (local address 127.0.0.1:57424)
0923 20:54:18.410307 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:18.410329 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:18.410514 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:18.411153 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:18.411157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:18.411173 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:54:18.411386 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.411395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.412573 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.412578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.413600 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.413613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.414491 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.414500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.414675 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.414743 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:54:18.415710 (+ 967us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:54:18.415733 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:54:18.415854 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 20:54:18.416019 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 20:54:18.421316 8446 raft_consensus.cc:1261] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Refusing update from remote peer 383c8e7c35f9483c815ff6817bc21c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:18.421681 8163 raft_consensus.cc:1261] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Refusing update from remote peer 383c8e7c35f9483c815ff6817bc21c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:18.422135 8547 consensus_queue.cc:1035] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:18.422588 8523 consensus_queue.cc:1035] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:18.427614 8522 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: TxnStatusTablet state changed. Reason: New leader 383c8e7c35f9483c815ff6817bc21c49. Latest consensus state: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.427964 8522 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:54:18.430907 8534 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: TxnStatusTablet state changed. Reason: New leader 383c8e7c35f9483c815ff6817bc21c49. Latest consensus state: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.431239 8534 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:54:18.435138 8547 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.435488 8547 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:54:18.436117 8523 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.436470 8523 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P 383c8e7c35f9483c815ff6817bc21c49: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:54:18.439589 8522 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.439836 8522 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P d06630c0491144b7ab6614f7adac8649: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:54:18.443091 8534 tablet_replica.cc:439] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } } }
I20260923 20:54:18.443342 8534 tablet_replica.cc:442] T 76a2e3f0d39b4cd8b65ed080c6a35e91 P e9809bbf187142ed8c59f10c4b5415ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:54:18.442616 8565 mvcc.cc:205] Tried to move back new op lower bound from 7332646332085645312 to 7332646331946078208. Current Snapshot: MvccSnapshot[applied={T|T < 7332646332085645312 or (T in {7332646332085645312})}]
I20260923 20:54:18.451478 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.446078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.0.97.2:38341)
0923 20:54:18.446250 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.446256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.446496 (+ 240us) server_negotiation.cc:407] Connection header received
0923 20:54:18.446706 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.446711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.446790 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.446887 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:18.447320 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.448172 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.449612 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.450212 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.450353 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.450753 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:18.450855 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:18.450967 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.451098 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.451182 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 20:54:18.469725 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.462962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42767 (local address 127.0.97.2:38341)
0923 20:54:18.463143 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.463148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.463891 (+ 743us) server_negotiation.cc:407] Connection header received
0923 20:54:18.464150 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.464155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.464264 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.464385 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:18.464942 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.466151 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.467646 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.468339 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.468480 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.468926 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:18.469063 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:18.469156 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.469283 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.469365 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:54:18.469775 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.462995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38341 (local address 127.0.97.2:42767)
0923 20:54:18.463775 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:18.463793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:18.463985 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:18.464392 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:18.464395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:18.464404 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:54:18.464785 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.464794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.466282 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.466287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.467497 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:18.467509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.468538 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.468542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:18.468682 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.468745 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:54:18.469164 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:54:18.469172 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:54:18.469237 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 20:54:18.469351 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":470,"threads_started":1}
I20260923 20:54:18.479898 8446 raft_consensus.cc:1261] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Refusing update from remote peer 383c8e7c35f9483c815ff6817bc21c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:18.479976 8162 raft_consensus.cc:1261] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Refusing update from remote peer 383c8e7c35f9483c815ff6817bc21c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:18.480463 8547 consensus_queue.cc:1035] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:18.480764 8523 consensus_queue.cc:1035] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:18.488646 8567 mvcc.cc:205] Tried to move back new op lower bound from 7332646332327297024 to 7332646331040968704. Current Snapshot: MvccSnapshot[applied={T|T < 7332646332327297024}]
I20260923 20:54:18.557574 8004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:18.557886 8004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:18.561348 8004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Sending DeleteTablet for 3 replicas of tablet e4834d774fc84caf93a9884fe468d5d7
I20260923 20:54:18.562263 8143 tablet_service.cc:1514] Processing DeleteTablet for tablet e4834d774fc84caf93a9884fe468d5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:54:18.562378 8421 tablet_service.cc:1514] Processing DeleteTablet for tablet e4834d774fc84caf93a9884fe468d5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54732
I20260923 20:54:18.562372 8281 tablet_service.cc:1514] Processing DeleteTablet for tablet e4834d774fc84caf93a9884fe468d5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57410
I20260923 20:54:18.573160 8581 tablet_replica.cc:330] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: stopping tablet replica
I20260923 20:54:18.573546 8581 raft_consensus.cc:2229] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:18.573902 8581 raft_consensus.cc:2258] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:18.574914 8583 tablet_replica.cc:330] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: stopping tablet replica
I20260923 20:54:18.575124 8583 raft_consensus.cc:2229] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:54:18.575769 8583 raft_consensus.cc:2258] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:18.577231 8583 ts_tablet_manager.cc:1905] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:18.578794 8581 ts_tablet_manager.cc:1905] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:18.579357 8582 tablet_replica.cc:330] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: stopping tablet replica
I20260923 20:54:18.579573 8582 raft_consensus.cc:2229] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:18.579834 8582 raft_consensus.cc:2258] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:18.582067 8582 ts_tablet_manager.cc:1905] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:18.586365 8583 ts_tablet_manager.cc:1918] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:54:18.586635 8583 log.cc:1196] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/e4834d774fc84caf93a9884fe468d5d7
I20260923 20:54:18.587203 8583 ts_tablet_manager.cc:1939] T e4834d774fc84caf93a9884fe468d5d7 P 383c8e7c35f9483c815ff6817bc21c49: Deleting consensus metadata
I20260923 20:54:18.588575 7993 catalog_manager.cc:4841] TS 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341): tablet e4834d774fc84caf93a9884fe468d5d7 (table test-table [id=f3faeb8d908849e7884fd18f1dac9c77]) successfully deleted
I20260923 20:54:18.589289 8582 ts_tablet_manager.cc:1918] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:54:18.589437 8582 log.cc:1196] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/e4834d774fc84caf93a9884fe468d5d7
I20260923 20:54:18.589918 8582 ts_tablet_manager.cc:1939] T e4834d774fc84caf93a9884fe468d5d7 P d06630c0491144b7ab6614f7adac8649: Deleting consensus metadata
I20260923 20:54:18.591130 7991 catalog_manager.cc:4841] TS d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875): tablet e4834d774fc84caf93a9884fe468d5d7 (table test-table [id=f3faeb8d908849e7884fd18f1dac9c77]) successfully deleted
I20260923 20:54:18.593878 8581 ts_tablet_manager.cc:1918] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:54:18.594020 8581 log.cc:1196] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/e4834d774fc84caf93a9884fe468d5d7
I20260923 20:54:18.594477 8581 ts_tablet_manager.cc:1939] T e4834d774fc84caf93a9884fe468d5d7 P e9809bbf187142ed8c59f10c4b5415ff: Deleting consensus metadata
I20260923 20:54:18.596261 7992 catalog_manager.cc:4841] TS e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723): tablet e4834d774fc84caf93a9884fe468d5d7 (table test-table [id=f3faeb8d908849e7884fd18f1dac9c77]) successfully deleted
Sep 23 20:54:18 dist-test-slave-plcg krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:54:18 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:18 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:54:18.640913 8555 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:18.629881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.0.97.62:38703)
0923 20:54:18.630079 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.630086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.630445 (+ 359us) server_negotiation.cc:407] Connection header received
0923 20:54:18.630613 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.630619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.630700 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.630819 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:18.632113 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.633281 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.634685 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.635356 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.635504 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.640662 (+ 5158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41538: BlockingRecv error: recv got EOF from 127.0.0.1:41538 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:54:18 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 20:54:18.659077 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.643738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.0.97.62:38703)
0923 20:54:18.643918 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.643925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.644237 (+ 312us) server_negotiation.cc:407] Connection header received
0923 20:54:18.644399 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.644405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.644484 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.644589 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:18.645689 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.646864 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.648131 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.648790 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.648929 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.652939 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:18.652946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:18.652950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:18.652995 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:18.656429 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:18.657122 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:18.657128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:18.657132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:18.657296 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:18.657655 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:18.657659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:18.657662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:18.658016 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:18.658266 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.658477 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.658704 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:54:18.662698 8004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:54:18.663421 8004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:18.670311 8143 tablet_service.cc:1467] Processing CreateTablet for tablet f6cb0c5781be45f1a7dcd60827b7356d (DEFAULT_TABLE table=test-table [id=2ae59569d14346368ebe66e516863f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:18.670639 8281 tablet_service.cc:1467] Processing CreateTablet for tablet f6cb0c5781be45f1a7dcd60827b7356d (DEFAULT_TABLE table=test-table [id=2ae59569d14346368ebe66e516863f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:18.670785 8143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cb0c5781be45f1a7dcd60827b7356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.671015 8281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cb0c5781be45f1a7dcd60827b7356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.674413 8421 tablet_service.cc:1467] Processing CreateTablet for tablet f6cb0c5781be45f1a7dcd60827b7356d (DEFAULT_TABLE table=test-table [id=2ae59569d14346368ebe66e516863f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:18.674865 8421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cb0c5781be45f1a7dcd60827b7356d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.676929 8515 tablet_bootstrap.cc:492] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: Bootstrap starting.
I20260923 20:54:18.678686 8515 tablet_bootstrap.cc:654] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.681563 8517 tablet_bootstrap.cc:492] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: Bootstrap starting.
I20260923 20:54:18.682773 8516 tablet_bootstrap.cc:492] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: Bootstrap starting.
I20260923 20:54:18.683334 8517 tablet_bootstrap.cc:654] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.684554 8516 tablet_bootstrap.cc:654] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.685901 8515 tablet_bootstrap.cc:492] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: No bootstrap required, opened a new log
I20260923 20:54:18.686081 8515 ts_tablet_manager.cc:1397] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260923 20:54:18.686973 8515 raft_consensus.cc:348] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.687213 8515 raft_consensus.cc:374] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.687301 8515 raft_consensus.cc:729] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.687558 8515 consensus_queue.cc:260] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.690615 8515 ts_tablet_manager.cc:1428] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 20:54:18.694772 8516 tablet_bootstrap.cc:492] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: No bootstrap required, opened a new log
I20260923 20:54:18.694972 8516 ts_tablet_manager.cc:1397] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20260923 20:54:18.695605 8516 raft_consensus.cc:348] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.695824 8516 raft_consensus.cc:374] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.695923 8516 raft_consensus.cc:729] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06630c0491144b7ab6614f7adac8649, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.696174 8516 consensus_queue.cc:260] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.696676 8516 ts_tablet_manager.cc:1428] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 20:54:18.698014 8517 tablet_bootstrap.cc:492] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: No bootstrap required, opened a new log
I20260923 20:54:18.698180 8517 ts_tablet_manager.cc:1397] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260923 20:54:18.699002 8517 raft_consensus.cc:348] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.699247 8517 raft_consensus.cc:374] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.699347 8517 raft_consensus.cc:729] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9809bbf187142ed8c59f10c4b5415ff, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.699612 8517 consensus_queue.cc:260] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.700753 8517 ts_tablet_manager.cc:1428] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:54:18.774999 8534 raft_consensus.cc:482] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:18.775243 8534 raft_consensus.cc:504] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.776119 8534 leader_election.cc:288] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:18.776613 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb0c5781be45f1a7dcd60827b7356d" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383c8e7c35f9483c815ff6817bc21c49" is_pre_election: true
I20260923 20:54:18.776814 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb0c5781be45f1a7dcd60827b7356d" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649" is_pre_election: true
I20260923 20:54:18.776919 8302 raft_consensus.cc:2454] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 0.
I20260923 20:54:18.777074 8446 raft_consensus.cc:2454] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 0.
I20260923 20:54:18.777422 8096 leader_election.cc:302] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, e9809bbf187142ed8c59f10c4b5415ff; no voters:
I20260923 20:54:18.777730 8534 raft_consensus.cc:2781] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:18.777871 8534 raft_consensus.cc:482] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:18.777961 8534 raft_consensus.cc:3037] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.780048 8534 raft_consensus.cc:504] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.780654 8534 leader_election.cc:288] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 election: Requested vote from peers 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:18.781214 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb0c5781be45f1a7dcd60827b7356d" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649"
I20260923 20:54:18.781191 8302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cb0c5781be45f1a7dcd60827b7356d" candidate_uuid: "e9809bbf187142ed8c59f10c4b5415ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383c8e7c35f9483c815ff6817bc21c49"
I20260923 20:54:18.781432 8446 raft_consensus.cc:3037] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.781441 8302 raft_consensus.cc:3037] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:18.783700 8446 raft_consensus.cc:2454] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 1.
I20260923 20:54:18.784171 8094 leader_election.cc:302] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d06630c0491144b7ab6614f7adac8649, e9809bbf187142ed8c59f10c4b5415ff; no voters:
I20260923 20:54:18.784447 8534 raft_consensus.cc:2781] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:18.784945 8534 raft_consensus.cc:686] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 1 LEADER]: Becoming Leader. State: Replica: e9809bbf187142ed8c59f10c4b5415ff, State: Running, Role: LEADER
I20260923 20:54:18.784919 8302 raft_consensus.cc:2454] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9809bbf187142ed8c59f10c4b5415ff in term 1.
I20260923 20:54:18.785537 8534 consensus_queue.cc:237] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.788584 8004 catalog_manager.cc:5495] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff reported cstate change: term changed from 0 to 1, leader changed from <none> to e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e9809bbf187142ed8c59f10c4b5415ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } health_report { overall_health: UNKNOWN } } }
I20260923 20:54:18.818599 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:18.811301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.0.97.1:38723)
0923 20:54:18.812202 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:18.812212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:18.812234 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:54:18.812304 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:18.812309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:18.812397 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:18.812499 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:18.812977 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.814239 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.816335 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:18.817055 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:18.817188 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:18.817710 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:18.817890 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:18.817975 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:18.818146 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 20:54:18.818232 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":603,"threads_started":1}
I20260923 20:54:18.821518 8446 raft_consensus.cc:1261] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Refusing update from remote peer e9809bbf187142ed8c59f10c4b5415ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:18.821759 8302 raft_consensus.cc:1261] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Refusing update from remote peer e9809bbf187142ed8c59f10c4b5415ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:18.822145 8599 consensus_queue.cc:1035] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:18.822582 8534 consensus_queue.cc:1035] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:18.848673 8006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:18.848946 8006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:18.852489 8006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Sending DeleteTablet for 3 replicas of tablet f6cb0c5781be45f1a7dcd60827b7356d
I20260923 20:54:18.853330 8143 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cb0c5781be45f1a7dcd60827b7356d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:54:18.853384 8281 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cb0c5781be45f1a7dcd60827b7356d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57410
I20260923 20:54:18.853693 8581 tablet_replica.cc:330] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: stopping tablet replica
I20260923 20:54:18.853708 8583 tablet_replica.cc:330] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: stopping tablet replica
I20260923 20:54:18.853940 8583 raft_consensus.cc:2229] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:18.854053 8581 raft_consensus.cc:2229] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:54:18.854206 8583 raft_consensus.cc:2258] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:18.854384 8581 raft_consensus.cc:2258] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:18.854403 8421 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cb0c5781be45f1a7dcd60827b7356d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54732
I20260923 20:54:18.854707 8582 tablet_replica.cc:330] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: stopping tablet replica
I20260923 20:54:18.854921 8582 raft_consensus.cc:2229] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:18.855152 8582 raft_consensus.cc:2258] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:18.855832 8583 ts_tablet_manager.cc:1905] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:18.856228 8581 ts_tablet_manager.cc:1905] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:18.856228 8582 ts_tablet_manager.cc:1905] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:18.860229 8005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:54:18.861064 8005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:18.863286 8581 ts_tablet_manager.cc:1918] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:18.863570 8581 log.cc:1196] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/f6cb0c5781be45f1a7dcd60827b7356d
I20260923 20:54:18.863551 8582 ts_tablet_manager.cc:1918] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:18.863693 8582 log.cc:1196] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/f6cb0c5781be45f1a7dcd60827b7356d
I20260923 20:54:18.864115 8582 ts_tablet_manager.cc:1939] T f6cb0c5781be45f1a7dcd60827b7356d P d06630c0491144b7ab6614f7adac8649: Deleting consensus metadata
I20260923 20:54:18.864454 8581 ts_tablet_manager.cc:1939] T f6cb0c5781be45f1a7dcd60827b7356d P e9809bbf187142ed8c59f10c4b5415ff: Deleting consensus metadata
I20260923 20:54:18.865329 7991 catalog_manager.cc:4841] TS d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875): tablet f6cb0c5781be45f1a7dcd60827b7356d (table test-table [id=2ae59569d14346368ebe66e516863f14]) successfully deleted
I20260923 20:54:18.866096 7992 catalog_manager.cc:4841] TS e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723): tablet f6cb0c5781be45f1a7dcd60827b7356d (table test-table [id=2ae59569d14346368ebe66e516863f14]) successfully deleted
I20260923 20:54:18.869966 8583 ts_tablet_manager.cc:1918] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:18.870138 8583 log.cc:1196] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/f6cb0c5781be45f1a7dcd60827b7356d
I20260923 20:54:18.870572 8583 ts_tablet_manager.cc:1939] T f6cb0c5781be45f1a7dcd60827b7356d P 383c8e7c35f9483c815ff6817bc21c49: Deleting consensus metadata
I20260923 20:54:18.870985 8143 tablet_service.cc:1467] Processing CreateTablet for tablet 5b82cc1a18ff4616bf456a58937ade90 (DEFAULT_TABLE table=test-table [id=06c28a7433db4a8e97f91dae3c9badde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:18.871357 8143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b82cc1a18ff4616bf456a58937ade90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.876817 8517 tablet_bootstrap.cc:492] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: Bootstrap starting.
I20260923 20:54:18.877102 7993 catalog_manager.cc:4841] TS 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341): tablet f6cb0c5781be45f1a7dcd60827b7356d (table test-table [id=2ae59569d14346368ebe66e516863f14]) successfully deleted
I20260923 20:54:18.878026 8421 tablet_service.cc:1467] Processing CreateTablet for tablet 5b82cc1a18ff4616bf456a58937ade90 (DEFAULT_TABLE table=test-table [id=06c28a7433db4a8e97f91dae3c9badde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:18.878249 8281 tablet_service.cc:1467] Processing CreateTablet for tablet 5b82cc1a18ff4616bf456a58937ade90 (DEFAULT_TABLE table=test-table [id=06c28a7433db4a8e97f91dae3c9badde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:18.878453 8421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b82cc1a18ff4616bf456a58937ade90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.878669 8281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b82cc1a18ff4616bf456a58937ade90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:18.878665 8517 tablet_bootstrap.cc:654] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.884614 8516 tablet_bootstrap.cc:492] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: Bootstrap starting.
I20260923 20:54:18.886451 8516 tablet_bootstrap.cc:654] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.889782 8515 tablet_bootstrap.cc:492] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: Bootstrap starting.
I20260923 20:54:18.890442 8517 tablet_bootstrap.cc:492] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: No bootstrap required, opened a new log
I20260923 20:54:18.890657 8517 ts_tablet_manager.cc:1397] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260923 20:54:18.891688 8515 tablet_bootstrap.cc:654] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:18.891769 8517 raft_consensus.cc:348] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.892100 8517 raft_consensus.cc:374] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.892191 8517 raft_consensus.cc:729] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9809bbf187142ed8c59f10c4b5415ff, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.892513 8517 consensus_queue.cc:260] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.894496 8517 ts_tablet_manager.cc:1428] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:54:18.899190 8516 tablet_bootstrap.cc:492] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: No bootstrap required, opened a new log
I20260923 20:54:18.899451 8516 ts_tablet_manager.cc:1397] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260923 20:54:18.900624 8516 raft_consensus.cc:348] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.901031 8516 raft_consensus.cc:374] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.901131 8516 raft_consensus.cc:729] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06630c0491144b7ab6614f7adac8649, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.901543 8516 consensus_queue.cc:260] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.902323 8516 ts_tablet_manager.cc:1428] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:18.906301 8515 tablet_bootstrap.cc:492] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: No bootstrap required, opened a new log
I20260923 20:54:18.906517 8515 ts_tablet_manager.cc:1397] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260923 20:54:18.907415 8515 raft_consensus.cc:348] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.907727 8515 raft_consensus.cc:374] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:18.907812 8515 raft_consensus.cc:729] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Initialized, Role: FOLLOWER
I20260923 20:54:18.908053 8515 consensus_queue.cc:260] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:18.908573 8515 ts_tablet_manager.cc:1428] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:54:19.301548 8523 raft_consensus.cc:482] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:19.301786 8523 raft_consensus.cc:504] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:19.302476 8523 leader_election.cc:288] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:19.303150 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82cc1a18ff4616bf456a58937ade90" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649" is_pre_election: true
I20260923 20:54:19.303138 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82cc1a18ff4616bf456a58937ade90" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9809bbf187142ed8c59f10c4b5415ff" is_pre_election: true
I20260923 20:54:19.303398 8446 raft_consensus.cc:2454] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 0.
I20260923 20:54:19.303450 8162 raft_consensus.cc:2454] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 0.
I20260923 20:54:19.303933 8234 leader_election.cc:302] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, d06630c0491144b7ab6614f7adac8649; no voters:
I20260923 20:54:19.304279 8523 raft_consensus.cc:2781] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:19.304394 8523 raft_consensus.cc:482] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:19.304538 8523 raft_consensus.cc:3037] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:19.306886 8523 raft_consensus.cc:504] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:19.307542 8523 leader_election.cc:288] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 election: Requested vote from peers e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723), d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875)
I20260923 20:54:19.308100 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82cc1a18ff4616bf456a58937ade90" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9809bbf187142ed8c59f10c4b5415ff"
I20260923 20:54:19.308125 8446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82cc1a18ff4616bf456a58937ade90" candidate_uuid: "383c8e7c35f9483c815ff6817bc21c49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d06630c0491144b7ab6614f7adac8649"
I20260923 20:54:19.308360 8446 raft_consensus.cc:3037] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:19.308375 8162 raft_consensus.cc:3037] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:19.311481 8162 raft_consensus.cc:2454] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 1.
I20260923 20:54:19.311506 8446 raft_consensus.cc:2454] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 383c8e7c35f9483c815ff6817bc21c49 in term 1.
I20260923 20:54:19.312013 8234 leader_election.cc:302] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383c8e7c35f9483c815ff6817bc21c49, d06630c0491144b7ab6614f7adac8649; no voters:
I20260923 20:54:19.312388 8523 raft_consensus.cc:2781] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:19.312809 8523 raft_consensus.cc:686] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 LEADER]: Becoming Leader. State: Replica: 383c8e7c35f9483c815ff6817bc21c49, State: Running, Role: LEADER
I20260923 20:54:19.313119 8523 consensus_queue.cc:237] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } }
I20260923 20:54:19.316026 8005 catalog_manager.cc:5495] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "383c8e7c35f9483c815ff6817bc21c49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "383c8e7c35f9483c815ff6817bc21c49" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 } health_report { overall_health: UNKNOWN } } }
I20260923 20:54:19.358872 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:19.353352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.97.2:38341)
0923 20:54:19.354226 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:19.354231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:19.354247 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:54:19.354311 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:19.354314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:19.354376 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:19.354489 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:19.354968 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.355691 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.357020 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.357472 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.357607 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:19.358081 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:19.358184 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:19.358285 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:19.358427 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 20:54:19.358502 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":609,"threads_started":1}
I20260923 20:54:19.368515 8162 raft_consensus.cc:1261] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Refusing update from remote peer 383c8e7c35f9483c815ff6817bc21c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:19.368515 8446 raft_consensus.cc:1261] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Refusing update from remote peer 383c8e7c35f9483c815ff6817bc21c49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:19.369127 8613 consensus_queue.cc:1035] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9809bbf187142ed8c59f10c4b5415ff" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:19.369644 8523 consensus_queue.cc:1035] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d06630c0491144b7ab6614f7adac8649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:19.420733 8005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:19.420959 8005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:19.423667 8005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec9cd7e252c143cd83a0b620c0c706a9: Sending DeleteTablet for 3 replicas of tablet 5b82cc1a18ff4616bf456a58937ade90
I20260923 20:54:19.424512 8143 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b82cc1a18ff4616bf456a58937ade90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:54:19.424512 8282 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b82cc1a18ff4616bf456a58937ade90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57410
I20260923 20:54:19.424588 8421 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b82cc1a18ff4616bf456a58937ade90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54732
I20260923 20:54:19.425274 8626 tablet_replica.cc:330] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: stopping tablet replica
I20260923 20:54:19.425478 8626 raft_consensus.cc:2229] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:19.425786 8626 raft_consensus.cc:2258] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:19.426337 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 8077
I20260923 20:54:19.428398 8626 ts_tablet_manager.cc:1905] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:19.437551 8626 ts_tablet_manager.cc:1918] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:54:19.437731 8626 log.cc:1196] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/5b82cc1a18ff4616bf456a58937ade90
I20260923 20:54:19.437960 8627 tablet_replica.cc:330] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: stopping tablet replica
I20260923 20:54:19.438293 8626 ts_tablet_manager.cc:1939] T 5b82cc1a18ff4616bf456a58937ade90 P e9809bbf187142ed8c59f10c4b5415ff: Deleting consensus metadata
I20260923 20:54:19.438266 8627 raft_consensus.cc:2229] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:19.438732 8627 raft_consensus.cc:2258] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:19.439766 7992 catalog_manager.cc:4841] TS e9809bbf187142ed8c59f10c4b5415ff (127.0.97.1:38723): tablet 5b82cc1a18ff4616bf456a58937ade90 (table test-table [id=06c28a7433db4a8e97f91dae3c9badde]) successfully deleted
I20260923 20:54:19.440151 8627 ts_tablet_manager.cc:1905] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:19.440765 8625 tablet_replica.cc:330] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: stopping tablet replica
I20260923 20:54:19.440971 8625 raft_consensus.cc:2229] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:54:19.441392 8625 raft_consensus.cc:2258] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:19.442270 8625 ts_tablet_manager.cc:1905] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:19.446496 8627 ts_tablet_manager.cc:1918] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:54:19.446652 8627 log.cc:1196] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/5b82cc1a18ff4616bf456a58937ade90
I20260923 20:54:19.447149 8627 ts_tablet_manager.cc:1939] T 5b82cc1a18ff4616bf456a58937ade90 P d06630c0491144b7ab6614f7adac8649: Deleting consensus metadata
I20260923 20:54:19.447796 8625 ts_tablet_manager.cc:1918] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:54:19.447953 8625 log.cc:1196] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/5b82cc1a18ff4616bf456a58937ade90
I20260923 20:54:19.448331 7991 catalog_manager.cc:4841] TS d06630c0491144b7ab6614f7adac8649 (127.0.97.3:36875): tablet 5b82cc1a18ff4616bf456a58937ade90 (table test-table [id=06c28a7433db4a8e97f91dae3c9badde]) successfully deleted
I20260923 20:54:19.448395 8625 ts_tablet_manager.cc:1939] T 5b82cc1a18ff4616bf456a58937ade90 P 383c8e7c35f9483c815ff6817bc21c49: Deleting consensus metadata
I20260923 20:54:19.449303 7993 catalog_manager.cc:4841] TS 383c8e7c35f9483c815ff6817bc21c49 (127.0.97.2:38341): tablet 5b82cc1a18ff4616bf456a58937ade90 (table test-table [id=06c28a7433db4a8e97f91dae3c9badde]) successfully deleted
Sep 23 20:54:19 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260923 20:54:19.461510 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:19.427572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.0.97.1:38723)
0923 20:54:19.439863 (+ 12291us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:19.439871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:19.439893 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:54:19.439949 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:19.439953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:19.440044 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:19.440228 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:19.442036 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.443248 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.445164 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.445878 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.446001 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:19.455409 (+ 9408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:19.455428 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:19.455447 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:19.455497 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:19.458588 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:19.459387 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:19.459395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:19.459397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:19.459548 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:19.459959 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:19.459964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:19.459966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:19.460322 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:19.460658 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:19.460923 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 20:54:19.461150 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12158,"thread_start_us":12504,"threads_started":1}
I20260923 20:54:19.462363 8208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:19.585348 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 8077
I20260923 20:54:19.608501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 8217
Sep 23 20:54:19 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260923 20:54:19.624106 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:19.609227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.0.97.2:38341)
0923 20:54:19.609416 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:19.609424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:19.609507 (+ 83us) server_negotiation.cc:407] Connection header received
0923 20:54:19.609707 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:19.609713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:19.609787 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:19.609908 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:19.611420 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.612134 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.613708 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.614212 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.614315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:19.618731 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:19.618747 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:19.618763 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:19.618813 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:19.621498 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:19.622334 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:19.622342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:19.622344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:19.622480 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:19.622782 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:19.622785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:19.622787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:19.623135 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:19.623378 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:19.623573 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 20:54:19.623765 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:54:19.624791 8347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:19.767351 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 8217
I20260923 20:54:19.792809 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 8356
Sep 23 20:54:19 dist-test-slave-plcg krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260923 20:54:19.808878 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:19.793555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.0.97.3:36875)
0923 20:54:19.794090 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:19.794101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:19.794118 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:54:19.794196 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:19.794199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:19.794258 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:19.794351 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:19.795568 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.796311 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.797490 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:19.798114 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:19.798214 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:19.803015 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:19.803030 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:19.803048 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:19.803084 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:19.805763 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:19.806647 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:19.806662 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:19.806664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:19.806791 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:19.807179 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:19.807186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:19.807189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:19.807628 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:19.807954 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:19.808261 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 20:54:19.808484 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":320,"threads_started":1}
I20260923 20:54:19.809540 8486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:19.913231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 8356
I20260923 20:54:19.930909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 7974
I20260923 20:54:19.932142 8046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:20.046608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 7974
2026-09-23T20:54:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6250 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:54:20 dist-test-slave-plcg krb5kdc[8637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:54:20 dist-test-slave-plcg krb5kdc[8637](info): set up 2 sockets
Sep 23 20:54:20 dist-test-slave-plcg krb5kdc[8637](info): commencing operation
krb5kdc: starting...
W20260923 20:54:21.142810 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:54:21 dist-test-slave-plcg krb5kdc[8637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:54:21.382189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34073
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:21.719071 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:21.719462 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:21.719635 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:21.727135 8650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:21.727309 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:21.727428 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:21.727540 8650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:21.727630 8650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:21.738747 8650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:21.740279 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:21.741734 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:21.754853 8655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:21.754813 8658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:21.754884 8656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:21.755654 8650 server_base.cc:1034] running on GCE node
I20260923 20:54:21.756553 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:21.758181 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:21.759426 8650 hybrid_clock.cc:648] HybridClock initialized: now 1790196861759361 us; error 66 us; skew 500 ppm
I20260923 20:54:21.759785 8650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T20:54:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1783 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:54:21 dist-test-slave-plcg krb5kdc[8664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:54:21 dist-test-slave-plcg krb5kdc[8664](info): set up 2 sockets
Sep 23 20:54:21 dist-test-slave-plcg krb5kdc[8664](info): commencing operation
krb5kdc: starting...
W20260923 20:54:22.939395 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:54:23 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:54:23.177197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35485
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:23.507953 8677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:23.508255 8677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:23.508349 8677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:23.515699 8677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:23.515831 8677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:23.515878 8677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:23.515929 8677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:23.515973 8677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:23.525966 8677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:23.527319 8677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:23.528597 8677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:23.541062 8685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:23.541086 8682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:23.540997 8683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:23.542443 8677 server_base.cc:1034] running on GCE node
I20260923 20:54:23.543310 8677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:23.544821 8677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:23.546025 8677 hybrid_clock.cc:648] HybridClock initialized: now 1790196863545973 us; error 43 us; skew 500 ppm
I20260923 20:54:23.546411 8677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:23 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:23.549801 8677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:54:23.551566 8677 webserver.cc:460] Webserver started at http://127.0.97.62:44533/ using document root <none> and password file <none>
I20260923 20:54:23.552331 8677 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:23.552487 8677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:23.552821 8677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:23.556088 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "63269631429748c791c7116ab2e47c1e"
format_stamp: "Formatted at 2026-09-23 20:54:23 on dist-test-slave-plcg"
server_key: "2dba58a437d4484c0207551c7c66e3a8"
server_key_iv: "019619d9baa47a7b3850a3e7f1eb5581"
server_key_version: "encryptionkey@0"
I20260923 20:54:23.556979 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "63269631429748c791c7116ab2e47c1e"
format_stamp: "Formatted at 2026-09-23 20:54:23 on dist-test-slave-plcg"
server_key: "2dba58a437d4484c0207551c7c66e3a8"
server_key_iv: "019619d9baa47a7b3850a3e7f1eb5581"
server_key_version: "encryptionkey@0"
I20260923 20:54:23.562665 8677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:54:23.566318 8692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:23.567708 8677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:54:23.567911 8677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "63269631429748c791c7116ab2e47c1e"
format_stamp: "Formatted at 2026-09-23 20:54:23 on dist-test-slave-plcg"
server_key: "2dba58a437d4484c0207551c7c66e3a8"
server_key_iv: "019619d9baa47a7b3850a3e7f1eb5581"
server_key_version: "encryptionkey@0"
I20260923 20:54:23.568192 8677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:23.596642 8677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:23.600996 8677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:23.601336 8677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:23.620231 8677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35485
I20260923 20:54:23.620261 8743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35485 every 8 connection(s)
I20260923 20:54:23.621701 8677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:23.625469 8744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:23.629534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 8677
I20260923 20:54:23.629745 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:54:23.630571 388 external_mini_cluster.cc:1428] Setting key 0790728e1dfe6266282d7f36564cc982
I20260923 20:54:23.637660 8744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Bootstrap starting.
I20260923 20:54:23.641490 8744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:23.642783 8744 log.cc:824] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:23.646519 8744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: No bootstrap required, opened a new log
Sep 23 20:54:23 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:23.654609 8744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:23.655043 8744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:23.655169 8744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63269631429748c791c7116ab2e47c1e, State: Initialized, Role: FOLLOWER
I20260923 20:54:23.655670 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:23.632783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.97.62:35485)
0923 20:54:23.633613 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:23.633624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:23.633666 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:54:23.636074 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:23.636102 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:23.636549 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:23.636912 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:23.638777 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:23.640163 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:23.642016 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:23.642735 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:23.642906 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:23.647608 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:23.647617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:23.647633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:23.647677 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:23.651157 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:23.651935 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:23.651942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:23.651950 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:23.652131 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:23.652541 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:23.652548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:23.652555 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:23.653164 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:23.653504 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:23.653942 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 20:54:23.654366 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":415,"threads_started":1}
I20260923 20:54:23.655784 8744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:23.656092 8744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:23.656253 8744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:23.656461 8744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:23.659382 8744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:23.659937 8744 leader_election.cc:302] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63269631429748c791c7116ab2e47c1e; no voters:
I20260923 20:54:23.660527 8744 leader_election.cc:288] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:54:23.660670 8749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:23.661339 8749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 LEADER]: Becoming Leader. State: Replica: 63269631429748c791c7116ab2e47c1e, State: Running, Role: LEADER
I20260923 20:54:23.661877 8749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:23.662665 8744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:23.664592 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63269631429748c791c7116ab2e47c1e. Latest consensus state: current_term: 1 leader_uuid: "63269631429748c791c7116ab2e47c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } } }
I20260923 20:54:23.665009 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:23.664938 8750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63269631429748c791c7116ab2e47c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } } }
I20260923 20:54:23.665251 8750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:23.668749 8754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:23.674499 8754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:23.685078 8754 catalog_manager.cc:1347] Generated new cluster ID: be0c42c142e54be28af511ab4a5fc4d1
I20260923 20:54:23.685225 8754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:23.704334 8754 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:54:23.705694 8754 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:23.716413 8754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Generated new TSK 0
I20260923 20:54:23.717545 8754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:54:23.781112 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35485
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:24.123929 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:24.124221 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:24.124317 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:24.131613 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:24.131733 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:24.131845 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:24.142000 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:24.143270 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:24.144582 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:24.158224 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:24.158195 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:24.158224 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:24.159860 8770 server_base.cc:1034] running on GCE node
I20260923 20:54:24.160523 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:24.161515 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:24.162748 8770 hybrid_clock.cc:648] HybridClock initialized: now 1790196864162689 us; error 87 us; skew 500 ppm
I20260923 20:54:24.163084 8770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:24 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:24.166430 8770 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:54:24.168036 8770 webserver.cc:460] Webserver started at http://127.0.97.1:43393/ using document root <none> and password file <none>
I20260923 20:54:24.168782 8770 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:24.168889 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:24.169163 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:24.172282 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "3364efdd8e84453fabb1ad50d28c4f62"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "e254a916c43eb6b8c514bba46eb38fec"
server_key_iv: "97cdb3f3ce8bb4f3c7c20f34eac862db"
server_key_version: "encryptionkey@0"
I20260923 20:54:24.173130 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3364efdd8e84453fabb1ad50d28c4f62"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "e254a916c43eb6b8c514bba46eb38fec"
server_key_iv: "97cdb3f3ce8bb4f3c7c20f34eac862db"
server_key_version: "encryptionkey@0"
I20260923 20:54:24.178676 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260923 20:54:24.182240 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:24.183605 8770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:54:24.183748 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "3364efdd8e84453fabb1ad50d28c4f62"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "e254a916c43eb6b8c514bba46eb38fec"
server_key_iv: "97cdb3f3ce8bb4f3c7c20f34eac862db"
server_key_version: "encryptionkey@0"
I20260923 20:54:24.183916 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:24.204370 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:24.208971 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:24.209236 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:24.210263 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:24.211644 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:24.211720 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:24.211814 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:24.211858 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:24.245528 8770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35229
I20260923 20:54:24.245570 8897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35229 every 8 connection(s)
I20260923 20:54:24.246928 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:24.253870 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 8770
I20260923 20:54:24.254099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:54:24.254747 388 external_mini_cluster.cc:1428] Setting key c87e833cee149c92ef3e918e4499a5c6
Sep 23 20:54:24 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:24.268455 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:24.249532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47019 (local address 127.0.97.62:35485)
0923 20:54:24.250024 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:24.250028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:24.251046 (+ 1018us) server_negotiation.cc:407] Connection header received
0923 20:54:24.252870 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:24.252875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:24.252977 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:24.253076 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:24.254896 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.255911 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.257542 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.258044 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.258188 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:24.262078 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:24.262088 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:24.262096 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:24.262144 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:24.265131 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:24.266016 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:24.266025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:24.266027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:24.266150 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:24.266628 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:24.266633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:24.266635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:24.266897 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:24.267066 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:24.267840 (+ 774us) server_negotiation.cc:299] Negotiation successful
0923 20:54:24.268088 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":288,"threads_started":1}
I20260923 20:54:24.269537 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:24.250073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:47019)
0923 20:54:24.250784 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:24.250836 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:24.252587 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:24.253317 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:24.253326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:24.253978 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:24.254678 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:24.254696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.256088 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.256094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:24.257378 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:24.257390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.258291 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.258300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:24.258558 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:24.259283 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:24.259311 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:24.261752 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:24.265315 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:24.265327 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:24.265350 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:24.265861 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:24.266270 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:24.266276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:24.266280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:24.266493 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:24.267041 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:24.267050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:24.267422 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 20:54:24.267717 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 20:54:24.268275 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
I20260923 20:54:24.271329 8898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35485
I20260923 20:54:24.271736 8898 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:24.272778 8898 heartbeater.cc:511] Master 127.0.97.62:35485 requested a full tablet report, sending...
I20260923 20:54:24.276188 8709 ts_manager.cc:194] Registered new tserver with Master: 3364efdd8e84453fabb1ad50d28c4f62 (127.0.97.1:35229)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:54:24.277799 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:54:24.307304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35485
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:24.652398 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:24.652743 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:24.652908 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:24.660648 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:24.660856 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:24.661060 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:24.671351 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:24.672721 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:24.674167 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:24.687978 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:24.688150 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:24.688087 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:24.689134 8904 server_base.cc:1034] running on GCE node
I20260923 20:54:24.689920 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:24.691004 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:24.692268 8904 hybrid_clock.cc:648] HybridClock initialized: now 1790196864692200 us; error 61 us; skew 500 ppm
I20260923 20:54:24.692655 8904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:24 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:24.702224 8904 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:54:24.703862 8904 webserver.cc:460] Webserver started at http://127.0.97.2:42503/ using document root <none> and password file <none>
I20260923 20:54:24.704610 8904 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:24.704758 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:24.705081 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:24.708205 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "9ad44fc00017466185f2adcdde19dd58"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "2a1accdd42c27f427a048bd2f32b4e64"
server_key_iv: "484236724bd3619fa1ca5df19c2cab7b"
server_key_version: "encryptionkey@0"
I20260923 20:54:24.709105 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9ad44fc00017466185f2adcdde19dd58"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "2a1accdd42c27f427a048bd2f32b4e64"
server_key_iv: "484236724bd3619fa1ca5df19c2cab7b"
server_key_version: "encryptionkey@0"
I20260923 20:54:24.714712 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.008s
I20260923 20:54:24.718722 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:24.720466 8904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260923 20:54:24.720678 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "9ad44fc00017466185f2adcdde19dd58"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "2a1accdd42c27f427a048bd2f32b4e64"
server_key_iv: "484236724bd3619fa1ca5df19c2cab7b"
server_key_version: "encryptionkey@0"
I20260923 20:54:24.720919 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:24.741057 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:24.746276 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:24.746620 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:24.747692 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:24.749255 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:24.749380 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:24.749538 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:24.749616 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:24.786695 8904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46599
I20260923 20:54:24.786794 9031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46599 every 8 connection(s)
I20260923 20:54:24.788240 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:24.790237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 8904
I20260923 20:54:24.790418 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:54:24.791052 388 external_mini_cluster.cc:1428] Setting key 0030e6f768e85568502ea1f8d901644e
Sep 23 20:54:24 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:24.813335 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:24.791240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46125 (local address 127.0.97.62:35485)
0923 20:54:24.791752 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:24.791760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:24.792892 (+ 1132us) server_negotiation.cc:407] Connection header received
0923 20:54:24.795904 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:24.795910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:24.796002 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:24.796115 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:24.798409 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.799209 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.801986 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.802552 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.802699 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:24.806841 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:24.806847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:24.806850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:24.806885 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:24.809650 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:24.810544 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:24.810555 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:24.810557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:24.810736 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:24.811384 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:24.811388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:24.811390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:24.811648 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:24.811759 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:24.812874 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0923 20:54:24.813050 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":263,"threads_started":1}
I20260923 20:54:24.815109 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:24.791937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:46125)
0923 20:54:24.792730 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:24.792778 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:24.795487 (+ 2709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:24.796436 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:24.796452 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:24.797270 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:24.798163 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:24.798191 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.799402 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.799410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:24.801759 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:24.801796 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:24.802806 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:24.802814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:24.803072 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:24.803893 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:24.803924 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:24.806544 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:24.809878 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:24.809889 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:24.809910 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:24.810389 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:24.810904 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:24.810913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:24.810919 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:24.811233 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:24.811804 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:24.811817 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:24.812398 (+ 581us) client_negotiation.cc:769] Sending connection context
0923 20:54:24.812795 (+ 397us) client_negotiation.cc:240] Negotiation successful
0923 20:54:24.813435 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:54:24.816787 9032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35485
I20260923 20:54:24.817087 9032 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:24.817934 9032 heartbeater.cc:511] Master 127.0.97.62:35485 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 20:54:24.819511 8709 ts_manager.cc:194] Registered new tserver with Master: 9ad44fc00017466185f2adcdde19dd58 (127.0.97.2:46599)
I20260923 20:54:24.820451 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:54:24.845448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35485
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:25.189955 9038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:25.190267 9038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:25.190450 9038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:25.197919 9038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:25.198042 9038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:25.198266 9038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:25.208174 9038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:25.209538 9038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:25.211094 9038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:25.224656 9043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:25.224634 9044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:25.224598 9046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:25.225485 9038 server_base.cc:1034] running on GCE node
I20260923 20:54:25.226202 9038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:25.227226 9038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:25.228471 9038 hybrid_clock.cc:648] HybridClock initialized: now 1790196865228405 us; error 66 us; skew 500 ppm
I20260923 20:54:25.228814 9038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:25 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:25.232034 9038 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:54:25.233646 9038 webserver.cc:460] Webserver started at http://127.0.97.3:38041/ using document root <none> and password file <none>
I20260923 20:54:25.234431 9038 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:25.234597 9038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:25.234916 9038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:25.237982 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "fd834a14b17f43079397f91bae1ebbe9"
format_stamp: "Formatted at 2026-09-23 20:54:25 on dist-test-slave-plcg"
server_key: "28b88b68b03a798c6ce5360b58410324"
server_key_iv: "f06a334984c59190949b67dc604da9ee"
server_key_version: "encryptionkey@0"
I20260923 20:54:25.238862 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fd834a14b17f43079397f91bae1ebbe9"
format_stamp: "Formatted at 2026-09-23 20:54:25 on dist-test-slave-plcg"
server_key: "28b88b68b03a798c6ce5360b58410324"
server_key_iv: "f06a334984c59190949b67dc604da9ee"
server_key_version: "encryptionkey@0"
I20260923 20:54:25.244436 9038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:54:25.248322 9053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:25.249928 9038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:54:25.250085 9038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "fd834a14b17f43079397f91bae1ebbe9"
format_stamp: "Formatted at 2026-09-23 20:54:25 on dist-test-slave-plcg"
server_key: "28b88b68b03a798c6ce5360b58410324"
server_key_iv: "f06a334984c59190949b67dc604da9ee"
server_key_version: "encryptionkey@0"
I20260923 20:54:25.250267 9038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:25.281901 8898 heartbeater.cc:503] Master 127.0.97.62:35485 was elected leader, sending a full tablet report...
I20260923 20:54:25.286326 9038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:25.290858 9038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:25.291131 9038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:25.292060 9038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:25.293418 9038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:25.293495 9038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:25.293589 9038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:25.293634 9038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:25.329317 9038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46123
I20260923 20:54:25.329375 9165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46123 every 8 connection(s)
I20260923 20:54:25.330699 9038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:25.340520 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9038
I20260923 20:54:25.340700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:54:25.341280 388 external_mini_cluster.cc:1428] Setting key 0292a1429a1053a646cf1c21726b290e
Sep 23 20:54:25 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:25.352058 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:25.333214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47749 (local address 127.0.97.62:35485)
0923 20:54:25.333653 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:25.333658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:25.334765 (+ 1107us) server_negotiation.cc:407] Connection header received
0923 20:54:25.336672 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:25.336677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:25.336780 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:25.336902 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:25.338626 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.339389 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.341705 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.342274 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.342419 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:25.346585 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:25.346591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:25.346594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:25.346628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:25.349159 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.350010 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.350014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.350016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.350144 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.350532 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.350536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.350538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.350796 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:25.350905 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:25.351579 (+ 674us) server_negotiation.cc:299] Negotiation successful
0923 20:54:25.351759 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260923 20:54:25.353109 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:25.333872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:47749)
0923 20:54:25.334596 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:25.334636 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:25.336344 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:25.337174 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:25.337184 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:25.337732 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:25.338399 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:25.338444 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.339577 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.339581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:25.341529 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:25.341549 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.342689 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.342706 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:25.343012 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:25.343875 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:25.343914 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:25.346368 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:25.349326 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:25.349334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:25.349349 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:25.349888 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:25.350237 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:25.350240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:25.350242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:25.350433 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:25.350902 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:25.350908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:25.351251 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 20:54:25.351495 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 20:54:25.351898 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
I20260923 20:54:25.354804 9166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35485
I20260923 20:54:25.355189 9166 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:25.356158 9166 heartbeater.cc:511] Master 127.0.97.62:35485 requested a full tablet report, sending...
I20260923 20:54:25.357681 8709 ts_manager.cc:194] Registered new tserver with Master: fd834a14b17f43079397f91bae1ebbe9 (127.0.97.3:46123)
I20260923 20:54:25.358521 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47749
I20260923 20:54:25.370076 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 20:54:25.514823 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 8770
Sep 23 20:54:25 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:54:25.532987 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:25.516081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.0.97.1:35229)
0923 20:54:25.516754 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:25.516761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:25.516787 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:54:25.516848 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:25.516854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:25.517083 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:25.517220 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:25.518910 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.519968 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.521367 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.522291 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.522407 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:25.526807 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:25.526823 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:25.526840 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:25.526881 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:25.529917 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.530735 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.530749 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.530751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.530891 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.531287 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.531292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.531296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.531671 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:25.531982 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:25.532312 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 20:54:25.532603 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":355,"threads_started":1}
I20260923 20:54:25.534420 8893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:25.618718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 8770
I20260923 20:54:25.640918 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 8904
Sep 23 20:54:25 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:54:25.657603 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:25.642038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.0.97.2:46599)
0923 20:54:25.642556 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:25.642563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:25.642588 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:54:25.642650 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:25.642657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:25.642864 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:25.643008 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:25.644221 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.645229 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.646670 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.647287 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.647433 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:25.651804 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:25.651819 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:25.651835 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:25.651874 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:25.655030 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.655835 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.655842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.655845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.655975 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.656293 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.656296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.656298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.656634 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:25.656850 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:25.657103 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 20:54:25.657298 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":253,"threads_started":1}
I20260923 20:54:25.658972 9027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:25.743695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 8904
I20260923 20:54:25.760128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9038
Sep 23 20:54:25 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:54:25.777711 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:25.761231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.0.97.3:46123)
0923 20:54:25.761735 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:25.761743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:25.761773 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:54:25.761907 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:25.761921 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:25.762144 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:25.762325 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:25.763709 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.764756 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.766141 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:25.766722 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:25.766872 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:25.771379 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:25.771394 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:25.771411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:25.771455 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:25.774667 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.775500 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.775513 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.775515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.775644 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:25.776042 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:25.776049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:25.776053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:25.776440 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:25.776761 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:25.777076 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 20:54:25.777348 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":227,"threads_started":1}
I20260923 20:54:25.779117 9161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:25.863354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9038
I20260923 20:54:25.879866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 8677
I20260923 20:54:25.881011 8739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:25.991060 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 8677
I20260923 20:54:26.011999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35485
--webserver_interface=127.0.97.62
--webserver_port=44533
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:26.366778 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:26.367156 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:26.367333 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:26.374796 9184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:26.374969 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:26.375099 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:26.375221 9184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:26.375308 9184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:26.385435 9184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:26.386870 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:26.388213 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:26.403503 9192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:26.403543 9190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:26.403589 9189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:26.404589 9184 server_base.cc:1034] running on GCE node
I20260923 20:54:26.405404 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:26.406926 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:26.408159 9184 hybrid_clock.cc:648] HybridClock initialized: now 1790196866408106 us; error 35 us; skew 500 ppm
I20260923 20:54:26.408478 9184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:26 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:26.411688 9184 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:54:26.413267 9184 webserver.cc:460] Webserver started at http://127.0.97.62:44533/ using document root <none> and password file <none>
I20260923 20:54:26.414024 9184 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:26.414184 9184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:26.419862 9184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:54:26.422991 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:26.424465 9184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:26.424667 9184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "63269631429748c791c7116ab2e47c1e"
format_stamp: "Formatted at 2026-09-23 20:54:23 on dist-test-slave-plcg"
server_key: "2dba58a437d4484c0207551c7c66e3a8"
server_key_iv: "019619d9baa47a7b3850a3e7f1eb5581"
server_key_version: "encryptionkey@0"
I20260923 20:54:26.425643 9184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:26.439415 9184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:26.443781 9184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:26.444092 9184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:26.463158 9184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35485
I20260923 20:54:26.463165 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35485 every 8 connection(s)
I20260923 20:54:26.464745 9184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:26.465355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9184
I20260923 20:54:26.466351 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35229
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35485
--webserver_port=43393
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:54:26.472146 9251 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:54:26.477768 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Bootstrap starting.
I20260923 20:54:26.507280 9251 log.cc:824] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:26.518996 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:54:26.519714 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Bootstrap complete.
I20260923 20:54:26.534210 9251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:26.535861 9251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63269631429748c791c7116ab2e47c1e, State: Initialized, Role: FOLLOWER
I20260923 20:54:26.536851 9251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:26.537233 9251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:26.537410 9251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:26.537662 9251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:54:26.543577 9251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:26.544267 9251 leader_election.cc:302] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63269631429748c791c7116ab2e47c1e; no voters:
I20260923 20:54:26.544967 9251 leader_election.cc:288] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:54:26.545153 9255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:54:26.546357 9255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [term 2 LEADER]: Becoming Leader. State: Replica: 63269631429748c791c7116ab2e47c1e, State: Running, Role: LEADER
I20260923 20:54:26.547024 9255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } }
I20260923 20:54:26.547189 9251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:26.549631 9256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "63269631429748c791c7116ab2e47c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } } }
I20260923 20:54:26.550333 9256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:26.549628 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63269631429748c791c7116ab2e47c1e. Latest consensus state: current_term: 2 leader_uuid: "63269631429748c791c7116ab2e47c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63269631429748c791c7116ab2e47c1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35485 } } }
I20260923 20:54:26.554025 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:26.567695 9263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:26.573622 9263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:26.575368 9263 catalog_manager.cc:1259] Loaded cluster ID: be0c42c142e54be28af511ab4a5fc4d1
I20260923 20:54:26.575560 9263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:26.579723 9263 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:26.580472 9272 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:54:26.581485 9263 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 63269631429748c791c7116ab2e47c1e: Loaded TSK: 0
I20260923 20:54:26.583158 9263 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:54:26.849087 9253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:26.849458 9253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:26.849627 9253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:26.857470 9253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:26.857673 9253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:26.857913 9253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:26.868818 9253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43393
--tserver_master_addrs=127.0.97.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:26.870256 9253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:26.871670 9253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:26.887216 9278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:26.887233 9277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:26.888283 9280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:26.888923 9253 server_base.cc:1034] running on GCE node
I20260923 20:54:26.889734 9253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:26.891024 9253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:26.892287 9253 hybrid_clock.cc:648] HybridClock initialized: now 1790196866892212 us; error 64 us; skew 500 ppm
I20260923 20:54:26.892705 9253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:26 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:26.897107 9253 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 20:54:26.899065 9253 webserver.cc:460] Webserver started at http://127.0.97.1:43393/ using document root <none> and password file <none>
I20260923 20:54:26.899999 9253 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:26.900157 9253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:26.907572 9253 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 20:54:26.911278 9287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:26.913085 9253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:54:26.913298 9253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "3364efdd8e84453fabb1ad50d28c4f62"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "e254a916c43eb6b8c514bba46eb38fec"
server_key_iv: "97cdb3f3ce8bb4f3c7c20f34eac862db"
server_key_version: "encryptionkey@0"
I20260923 20:54:26.914858 9253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:26.934959 9253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:26.940459 9253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:26.940763 9253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:26.942042 9253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:26.943857 9253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:26.943974 9253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:26.944092 9253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:26.944178 9253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:26.980115 9253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35229
I20260923 20:54:26.980209 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35229 every 8 connection(s)
I20260923 20:54:26.981650 9253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:26.983924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9253
I20260923 20:54:26.984594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46599
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35485
--webserver_port=42503
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:54:26 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:27.005424 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:26.985197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:44413)
0923 20:54:26.986060 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:26.986122 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:26.987834 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:26.991814 (+ 3980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:26.991828 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:26.992624 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:26.993445 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:26.993476 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:26.996237 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:26.996242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:26.997555 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:26.997572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:26.999316 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:26.999322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:26.999531 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.000308 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.000339 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.004389 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":351,"threads_started":1}
W20260923 20:54:27.006276 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:26.985184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44413 (local address 127.0.97.62:35485)
0923 20:54:26.990342 (+ 5158us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:26.990358 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:26.990403 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:54:26.990767 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:26.990783 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:26.991186 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:26.991621 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:26.993776 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:26.996013 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:26.997761 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:26.999039 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:26.999258 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.004929 (+ 5671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44413: BlockingRecv error: recv got EOF from 127.0.97.1:44413 (error 108)
Metrics: {"server-negotiator.queue_time_us":4924,"thread_start_us":4711,"threads_started":1}
W20260923 20:54:27.006354 9400 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35485 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35485: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:27.017103 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.007241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:41155)
0923 20:54:27.007464 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.007486 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.007618 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.007946 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.007953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.008453 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.008687 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.008703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.009905 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.009912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.011614 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.011632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.012451 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.012459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.012637 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.013314 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.013338 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.016845 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 20:54:27.017426 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.007476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41155 (local address 127.0.97.62:35485)
0923 20:54:27.007633 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.007638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.007656 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:54:27.007734 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.007738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.007795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.007874 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.008910 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.009718 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.011813 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.012287 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.012414 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.017170 (+ 4756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41155: BlockingRecv error: recv got EOF from 127.0.97.1:41155 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:27.027390 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.018124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:45763)
0923 20:54:27.018317 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.018336 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.018445 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.018799 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.018804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.019207 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.019401 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.019412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.020547 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.020553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.022186 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.022207 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.022950 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.022955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.023151 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.023796 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.023819 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.027147 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 20:54:27.027688 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.018277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45763 (local address 127.0.97.62:35485)
0923 20:54:27.018427 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.018431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.018458 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:54:27.018578 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.018582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.018660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.018753 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.019604 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.020399 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.022376 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.022796 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.022900 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.027435 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45763: BlockingRecv error: recv got EOF from 127.0.97.1:45763 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 20:54:27.027779 9400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:54:27.336462 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:27.336779 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:27.336936 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:27.344645 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:27.344779 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:27.344966 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:27.355290 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42503
--tserver_master_addrs=127.0.97.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:27.356735 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:27.358129 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:27.372603 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:27.372615 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:27.372615 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:27.373447 9404 server_base.cc:1034] running on GCE node
I20260923 20:54:27.374260 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:27.375321 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:27.376580 9404 hybrid_clock.cc:648] HybridClock initialized: now 1790196867376529 us; error 59 us; skew 500 ppm
I20260923 20:54:27.376972 9404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:27.380530 9404 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 20:54:27.382297 9404 webserver.cc:460] Webserver started at http://127.0.97.2:42503/ using document root <none> and password file <none>
I20260923 20:54:27.383061 9404 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:27.383225 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:27.389020 9404 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:54:27.392168 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:27.393654 9404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:54:27.393929 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "9ad44fc00017466185f2adcdde19dd58"
format_stamp: "Formatted at 2026-09-23 20:54:24 on dist-test-slave-plcg"
server_key: "2a1accdd42c27f427a048bd2f32b4e64"
server_key_iv: "484236724bd3619fa1ca5df19c2cab7b"
server_key_version: "encryptionkey@0"
I20260923 20:54:27.394982 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:27.419826 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:27.424132 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:27.424422 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:27.425436 9404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:27.426968 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:27.427110 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:27.427234 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:27.427338 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:27.459968 9404 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46599
I20260923 20:54:27.460027 9531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46599 every 8 connection(s)
I20260923 20:54:27.461418 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:27.462209 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9404
I20260923 20:54:27.462895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46123
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35485
--webserver_port=38041
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:54:27.487857 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.466851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33983 (local address 127.0.97.62:35485)
0923 20:54:27.467074 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.467083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.471473 (+ 4390us) server_negotiation.cc:407] Connection header received
0923 20:54:27.474239 (+ 2766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.474248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.474341 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.474452 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.476718 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.477968 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.480700 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.481145 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.481260 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.487601 (+ 6341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33983: BlockingRecv error: recv got EOF from 127.0.97.2:33983 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:54:27.488683 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.466026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:33983)
0923 20:54:27.471274 (+ 5248us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.471335 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.473915 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.474893 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.474909 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.475692 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.476495 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.476523 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.478232 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.478241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.480519 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.480541 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.481435 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.481445 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.481800 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.482671 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.482713 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.487371 (+ 4658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4900,"thread_start_us":4656,"threads_started":1}
W20260923 20:54:27.489462 9532 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35485 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35485: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:27.498407 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.490212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:48645)
0923 20:54:27.490425 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.490446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.490591 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.490938 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.490942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.491305 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.491448 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.491456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.492538 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.492543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.493457 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.493469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.494269 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.494275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.494459 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.495144 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.495163 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.498097 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 20:54:27.498567 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.490299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48645 (local address 127.0.97.62:35485)
0923 20:54:27.490486 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.490494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.490516 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:54:27.490726 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.490731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.490803 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.490911 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.491612 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.492378 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.493622 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.494098 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.494253 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.498325 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48645: BlockingRecv error: recv got EOF from 127.0.97.2:48645 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:27.507238 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.499342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:46025)
0923 20:54:27.499526 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.499547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.499664 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.500010 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.500014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.500349 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.500499 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.500517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.501529 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.501534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.502548 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.502560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.503253 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.503259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.503434 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.504064 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.504082 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.506921 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 20:54:27.507388 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.499440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46025 (local address 127.0.97.62:35485)
0923 20:54:27.499611 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.499617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.499635 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:54:27.499773 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.499778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.499852 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.499985 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.500654 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.501374 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.502691 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.503104 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.503241 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.507152 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46025: BlockingRecv error: recv got EOF from 127.0.97.2:46025 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 20:54:27.507709 9532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:54:27.815074 9535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:27.815369 9535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:27.815536 9535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:27.823237 9535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:27.823354 9535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:27.823534 9535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:27.833847 9535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38041
--tserver_master_addrs=127.0.97.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:27.835260 9535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:27.836606 9535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:27.850595 9540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:27.850638 9541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:27.850674 9543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:27.851647 9535 server_base.cc:1034] running on GCE node
I20260923 20:54:27.852362 9535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:27.853374 9535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:27.854614 9535 hybrid_clock.cc:648] HybridClock initialized: now 1790196867854540 us; error 91 us; skew 500 ppm
I20260923 20:54:27.854967 9535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:27.858387 9535 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 20:54:27.860167 9535 webserver.cc:460] Webserver started at http://127.0.97.3:38041/ using document root <none> and password file <none>
I20260923 20:54:27.860963 9535 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:27.861127 9535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:27.866927 9535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:54:27.869752 9550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:27.871289 9535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:54:27.871510 9535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "fd834a14b17f43079397f91bae1ebbe9"
format_stamp: "Formatted at 2026-09-23 20:54:25 on dist-test-slave-plcg"
server_key: "28b88b68b03a798c6ce5360b58410324"
server_key_iv: "f06a334984c59190949b67dc604da9ee"
server_key_version: "encryptionkey@0"
I20260923 20:54:27.872614 9535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:27.884950 9535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:27.891479 9535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:27.891863 9535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:27.893168 9535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:27.894908 9535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:27.895057 9535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:27.895196 9535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:27.895296 9535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:27.930121 9535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46123
I20260923 20:54:27.930303 9662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46123 every 8 connection(s)
I20260923 20:54:27.931701 9535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:27.933838 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9535
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:54:27.951272 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.934558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35389 (local address 127.0.97.62:35485)
0923 20:54:27.934757 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.934778 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.936605 (+ 1827us) server_negotiation.cc:407] Connection header received
0923 20:54:27.940246 (+ 3641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.940251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.940354 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.940451 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.942292 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.943230 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.945038 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.945585 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.945715 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.950964 (+ 5249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35389: BlockingRecv error: recv got EOF from 127.0.97.3:35389 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":70}
I20260923 20:54:27.951995 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.935288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:35389)
0923 20:54:27.936360 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.936434 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.939785 (+ 3351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.940820 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.940830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.941397 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.942109 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.942130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.943390 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.943396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.944887 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.944900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.945851 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.945860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.946135 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.946847 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.946882 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.950771 (+ 3889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":397,"threads_started":1}
W20260923 20:54:27.952850 9663 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35485 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35485: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260923 20:54:27.955940 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.934794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.0.97.62:35485)
0923 20:54:27.940854 (+ 6060us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.940858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.940872 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:54:27.940907 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.940910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.940952 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.941122 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.942856 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.943886 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.945151 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.945585 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.945730 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.948943 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:27.948960 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:27.948987 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:27.949045 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:27.952386 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:27.953338 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:27.953350 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:27.953362 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:27.953593 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:27.954019 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:27.954027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:27.954030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:27.954478 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:27.954831 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:27.955246 (+ 415us) server_negotiation.cc:299] Negotiation successful
0923 20:54:27.955631 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":5967,"thread_start_us":5883,"threads_started":1}
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:27.962221 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.953500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:55709)
0923 20:54:27.953671 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.953692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.953966 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.954295 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.954299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.954646 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.954794 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.954802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.955909 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.955914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.956972 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.956986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.957760 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.957765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.957992 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.958784 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.958803 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.961919 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 20:54:27.962422 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.953661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55709 (local address 127.0.97.62:35485)
0923 20:54:27.953796 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.953800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.953915 (+ 115us) server_negotiation.cc:407] Connection header received
0923 20:54:27.954101 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.954112 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.954168 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.954271 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.954934 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.955775 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.957138 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.957603 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.957736 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.962164 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55709: BlockingRecv error: recv got EOF from 127.0.97.3:55709 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":37}
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:27 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:27.970737 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:27.963009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:46097)
0923 20:54:27.963192 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:27.963212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:27.963323 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:27.963663 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:27.963670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:27.963992 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:27.964141 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.964150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.965189 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.965194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.966211 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:27.966223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.966887 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.966893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:27.967072 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.967672 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:27.967689 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:27.970406 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 20:54:27.971027 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:27.963147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46097 (local address 127.0.97.62:35485)
0923 20:54:27.963380 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:27.963389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:27.963406 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:54:27.963468 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:27.963472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:27.963525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:27.963630 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:27.964287 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.965035 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.966354 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:27.966742 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:27.966876 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:27.970602 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46097: BlockingRecv error: recv got EOF from 127.0.97.3:46097 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260923 20:54:27.971191 9663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 20:54:28 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:28 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:28.037313 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:28.028418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:50179)
0923 20:54:28.028986 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:28.029005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:28.029193 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:28.029560 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:28.029575 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:28.030110 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:28.030301 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:28.030309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.031336 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.031342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:28.032330 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:28.032343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.033313 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.033322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:28.033498 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:28.034216 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:28.034234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:28.037076 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":283,"threads_started":1}
W20260923 20:54:28.037542 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:28.028603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50179 (local address 127.0.97.62:35485)
0923 20:54:28.028834 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:28.028841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:28.029116 (+ 275us) server_negotiation.cc:407] Connection header received
0923 20:54:28.029319 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:28.029325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:28.029396 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:28.029506 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:28.030455 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.031188 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.032489 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.033042 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.033198 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:28.037298 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50179: BlockingRecv error: recv got EOF from 127.0.97.1:50179 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 20:54:28 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:28 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:28.517395 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:28.508445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:50417)
0923 20:54:28.508985 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:28.509006 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:28.509120 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:28.509468 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:28.509478 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:28.510104 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:28.510303 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:28.510312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.511360 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.511365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:28.512419 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:28.512432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.513266 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.513271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:28.513403 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:28.514224 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:28.514243 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:28.517166 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":288,"threads_started":1}
W20260923 20:54:28.517733 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:28.508510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50417 (local address 127.0.97.62:35485)
0923 20:54:28.508779 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:28.508785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:28.509105 (+ 320us) server_negotiation.cc:407] Connection header received
0923 20:54:28.509216 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:28.509222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:28.509312 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:28.509396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:28.510462 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.511191 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.512608 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.513036 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.513165 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:28.517416 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50417: BlockingRecv error: recv got EOF from 127.0.97.2:50417 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 20:54:28 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:28 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:28.980974 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:28.971913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:58179)
0923 20:54:28.972415 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:28.972435 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:28.972595 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:28.972957 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:28.972962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:28.973446 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:28.973633 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:28.973642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.974861 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.974867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:28.975864 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:28.975875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.976641 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.976647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:28.976828 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:28.977447 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:28.977471 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:28.980751 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":291,"threads_started":1}
W20260923 20:54:28.981242 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:28.971955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58179 (local address 127.0.97.62:35485)
0923 20:54:28.972182 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:28.972190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:28.972521 (+ 331us) server_negotiation.cc:407] Connection header received
0923 20:54:28.972727 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:28.972733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:28.972800 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:28.972914 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:28.973780 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.974678 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.976012 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:28.976427 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:28.976598 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:28.980982 (+ 4384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58179: BlockingRecv error: recv got EOF from 127.0.97.3:58179 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 20:54:29 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:29 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:29.047529 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:29.038434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:48317)
0923 20:54:29.038938 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:29.038960 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:29.039094 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:29.039447 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:29.039452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:29.040005 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:29.040217 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:29.040232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.041272 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.041277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:29.042218 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:29.042231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.042958 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.042964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:29.043133 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:29.043731 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:29.043751 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:29.047172 (+ 3421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":306,"threads_started":1}
W20260923 20:54:29.047698 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:29.038461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48317 (local address 127.0.97.62:35485)
0923 20:54:29.038675 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:29.038684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:29.039054 (+ 370us) server_negotiation.cc:407] Connection header received
0923 20:54:29.039216 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:29.039220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:29.039283 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:29.039409 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:29.040370 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.041114 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.042377 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.042791 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.042935 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:29.047440 (+ 4505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48317: BlockingRecv error: recv got EOF from 127.0.97.1:48317 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 20:54:29 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:29 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:29.527642 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:29.518614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:50671)
0923 20:54:29.519170 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:29.519191 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:29.519363 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:29.519720 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:29.519725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:29.520188 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:29.520338 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:29.520348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.521776 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.521781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:29.522978 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:29.523013 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.523752 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.523757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:29.523869 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:29.524498 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:29.524518 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:29.527394 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":307,"threads_started":1}
W20260923 20:54:29.528128 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:29.518700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50671 (local address 127.0.97.62:35485)
0923 20:54:29.518951 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:29.518960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:29.519287 (+ 327us) server_negotiation.cc:407] Connection header received
0923 20:54:29.519494 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:29.519507 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:29.519570 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:29.519681 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:29.520519 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.521607 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.523159 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.523597 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.523704 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:29.527632 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50671: BlockingRecv error: recv got EOF from 127.0.97.2:50671 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 23 20:54:29 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:29 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:29.995437 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:29.982048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:42635)
0923 20:54:29.982603 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:29.982625 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:29.982739 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:29.983109 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:29.983112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:29.983513 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:29.983655 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:29.983663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.984705 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.984709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:29.985614 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:29.985626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.986363 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.986367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:29.986482 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:29.991718 (+ 5236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:29.991745 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:29.995167 (+ 3422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":320,"threads_started":1}
W20260923 20:54:29.995618 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:29.982202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42635 (local address 127.0.97.62:35485)
0923 20:54:29.982407 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:29.982415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:29.982715 (+ 300us) server_negotiation.cc:407] Connection header received
0923 20:54:29.982854 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:29.982860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:29.982958 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:29.983083 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:29.983809 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.984566 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.985745 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:29.986216 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:29.986321 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:29.995375 (+ 9054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42635: BlockingRecv error: recv got EOF from 127.0.97.3:42635 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 20:54:30 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:30 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:30.057489 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:30.048739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:55787)
0923 20:54:30.049240 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:30.049261 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:30.049387 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:30.049712 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:30.049715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:30.050114 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:30.050294 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:30.050302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.051383 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.051389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:30.052312 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:30.052324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.053032 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.053037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:30.053262 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:30.054074 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:30.054094 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:30.057238 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
W20260923 20:54:30.057695 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:30.048850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55787 (local address 127.0.97.62:35485)
0923 20:54:30.049098 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:30.049105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:30.049353 (+ 248us) server_negotiation.cc:407] Connection header received
0923 20:54:30.049502 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:30.049506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:30.049573 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:30.049683 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:30.050460 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.051222 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.052461 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.052872 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.052982 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:30.057441 (+ 4459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55787: BlockingRecv error: recv got EOF from 127.0.97.1:55787 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 20:54:30 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:30 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:30.537379 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:30.528610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:36987)
0923 20:54:30.529117 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:30.529141 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:30.529322 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:30.529798 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:30.529802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:30.530241 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:30.530414 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:30.530424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.531585 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.531589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:30.532510 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:30.532523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.533245 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.533250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:30.533380 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:30.534080 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:30.534099 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:30.537121 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":296,"threads_started":1}
W20260923 20:54:30.537628 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:30.528731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36987 (local address 127.0.97.62:35485)
0923 20:54:30.528918 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:30.528923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:30.529263 (+ 340us) server_negotiation.cc:407] Connection header received
0923 20:54:30.529551 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:30.529556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:30.529619 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:30.529712 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:30.530617 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.531418 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.532659 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.533096 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.533193 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:30.537380 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36987: BlockingRecv error: recv got EOF from 127.0.97.2:36987 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 20:54:31 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:31 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:31.005957 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:30.996670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:45607)
0923 20:54:30.997218 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:30.997250 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:30.997380 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:30.997726 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:30.997730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:30.998162 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:30.998328 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:30.998338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.999498 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:30.999504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:31.000916 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:31.000933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.001738 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.001742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:31.001954 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:31.002628 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:31.002646 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:31.005552 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":323,"threads_started":1}
W20260923 20:54:31.006067 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:30.996766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45607 (local address 127.0.97.62:35485)
0923 20:54:30.996947 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:30.996952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:30.997391 (+ 439us) server_negotiation.cc:407] Connection header received
0923 20:54:30.997474 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:30.997478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:30.997541 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:30.997634 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:30.998505 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:30.999337 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.001059 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.001605 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.001702 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:31.005781 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45607: BlockingRecv error: recv got EOF from 127.0.97.3:45607 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 20:54:31 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:31 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:31.067530 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:31.058516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:49251)
0923 20:54:31.059197 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:31.059216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:31.059391 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:31.059734 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:31.059739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:31.060148 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:31.060294 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:31.060303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.061353 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.061361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:31.062370 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:31.062382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.063315 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.063321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:31.063524 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:31.064245 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:31.064279 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:31.067220 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":398,"threads_started":1}
W20260923 20:54:31.067852 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:31.058766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49251 (local address 127.0.97.62:35485)
0923 20:54:31.059110 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:31.059117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:31.059314 (+ 197us) server_negotiation.cc:407] Connection header received
0923 20:54:31.059532 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:31.059536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:31.059597 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:31.059706 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:31.060462 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.061204 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.062533 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.063150 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.063299 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:31.067573 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49251: BlockingRecv error: recv got EOF from 127.0.97.1:49251 (error 108)
Metrics: {"server-negotiator.queue_time_us":203}
Sep 23 20:54:31 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:31 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:31.547766 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:31.538490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:35021)
0923 20:54:31.539000 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:31.539020 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:31.539146 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:31.539527 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:31.539531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:31.539915 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:31.540202 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:31.540213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.541316 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.541321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:31.542414 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:31.542430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.543247 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.543251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:31.543388 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:31.544170 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:31.544189 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:31.547538 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":283,"threads_started":1}
W20260923 20:54:31.547971 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:31.538711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35021 (local address 127.0.97.62:35485)
0923 20:54:31.538888 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:31.538892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:31.539151 (+ 259us) server_negotiation.cc:407] Connection header received
0923 20:54:31.539234 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:31.539238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:31.539327 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:31.539417 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:31.540401 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.541130 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.542606 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:31.543034 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:31.543141 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:31.547725 (+ 4584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35021: BlockingRecv error: recv got EOF from 127.0.97.2:35021 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:54:32 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:32 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:32.017066 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:32.007069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:33375)
0923 20:54:32.007667 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:32.007689 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:32.007881 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:32.008314 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:32.008320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:32.008870 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:32.009103 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:32.009117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.010687 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.010706 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:32.011811 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:32.011827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.012575 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.012580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:32.012726 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:32.013433 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:32.013452 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:32.016791 (+ 3339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":354,"threads_started":1}
W20260923 20:54:32.017309 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:32.007202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33375 (local address 127.0.97.62:35485)
0923 20:54:32.007479 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:32.007489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:32.007807 (+ 318us) server_negotiation.cc:407] Connection header received
0923 20:54:32.008032 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:32.008044 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:32.008114 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:32.008233 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:32.009298 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.010326 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.012000 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.012408 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.012518 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:32.017024 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33375: BlockingRecv error: recv got EOF from 127.0.97.3:33375 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 23 20:54:32 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:32 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:32.082341 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:32.069309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:55571)
0923 20:54:32.072125 (+ 2816us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:32.072164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:32.072451 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:32.073051 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:32.073055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:32.073662 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:32.073930 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:32.073949 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.075729 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.075736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:32.076712 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:32.076727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.077495 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.077501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:32.077678 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:32.078419 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:32.078438 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:32.081976 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2542,"thread_start_us":2267,"threads_started":1}
W20260923 20:54:32.082435 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:32.070095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55571 (local address 127.0.97.62:35485)
0923 20:54:32.070534 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:32.070544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:32.072307 (+ 1763us) server_negotiation.cc:407] Connection header received
0923 20:54:32.072620 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:32.072627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:32.072712 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:32.072798 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:32.074202 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.075531 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.076875 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.077320 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.077478 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:32.082207 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55571: BlockingRecv error: recv got EOF from 127.0.97.1:55571 (error 108)
Metrics: {"server-negotiator.queue_time_us":263}
Sep 23 20:54:32 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:32 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:32.556553 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:32.548688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:55449)
0923 20:54:32.549179 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:32.549199 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:32.549307 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:32.549580 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:32.549584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:32.550028 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:32.550193 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:32.550202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.551198 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.551201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:32.552118 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:32.552130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.552816 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.552820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:32.552939 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:32.553543 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:32.553560 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:32.556301 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":248,"threads_started":1}
W20260923 20:54:32.556828 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:32.548890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55449 (local address 127.0.97.62:35485)
0923 20:54:32.549114 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:32.549123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:32.549275 (+ 152us) server_negotiation.cc:407] Connection header received
0923 20:54:32.549392 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:32.549396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:32.549456 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:32.549537 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:32.550322 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.551079 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.552263 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:32.552687 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:32.552780 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:32.556544 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55449: BlockingRecv error: recv got EOF from 127.0.97.2:55449 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 23 20:54:33 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:33 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:33.027117 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:33.018221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:40099)
0923 20:54:33.018769 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:33.018792 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:33.018946 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:33.019222 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:33.019224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:33.019666 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:33.019873 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:33.019884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.020888 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.020892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:33.022205 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:33.022219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.022933 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.022938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:33.023067 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:33.023698 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:33.023717 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:33.026853 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":290,"threads_started":1}
W20260923 20:54:33.027405 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:33.018261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40099 (local address 127.0.97.62:35485)
0923 20:54:33.018420 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:33.018424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:33.018896 (+ 472us) server_negotiation.cc:407] Connection header received
0923 20:54:33.019055 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:33.019059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:33.019123 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:33.019216 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:33.020033 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.020761 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.022364 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.022796 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.022898 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:33.027114 (+ 4216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40099: BlockingRecv error: recv got EOF from 127.0.97.3:40099 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 20:54:33 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:33 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:33.092060 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:33.083283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:44697)
0923 20:54:33.083762 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:33.083783 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:33.083901 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:33.084237 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:33.084241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:33.084630 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:33.084782 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:33.084790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.085932 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.085937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:33.086857 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:33.086869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.087644 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.087648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:33.087796 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:33.088439 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:33.088460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:33.091736 (+ 3276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":278,"threads_started":1}
W20260923 20:54:33.092299 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:33.083360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44697 (local address 127.0.97.62:35485)
0923 20:54:33.083633 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:33.083639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:33.083867 (+ 228us) server_negotiation.cc:407] Connection header received
0923 20:54:33.084005 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:33.084009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:33.084080 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:33.084166 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:33.084928 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.085641 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.087033 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.087485 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.087591 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:33.091983 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44697: BlockingRecv error: recv got EOF from 127.0.97.1:44697 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 20:54:33 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:33 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:33.566267 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:33.557712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:33309)
0923 20:54:33.558294 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:33.558322 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:33.558481 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:33.558860 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:33.558865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:33.559402 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:33.559604 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:33.559650 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.560766 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.560770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:33.561675 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:33.561687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.562529 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.562534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:33.562658 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:33.563230 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:33.563248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:33.566044 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":318,"threads_started":1}
W20260923 20:54:33.566584 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:33.557806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33309 (local address 127.0.97.62:35485)
0923 20:54:33.558076 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:33.558084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:33.558449 (+ 365us) server_negotiation.cc:407] Connection header received
0923 20:54:33.558616 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:33.558622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:33.558702 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:33.558798 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:33.559807 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.560630 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.561866 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:33.562377 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:33.562480 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:33.566271 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33309: BlockingRecv error: recv got EOF from 127.0.97.2:33309 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 20:54:34 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:34 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:34.036227 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:34.028113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:49931)
0923 20:54:34.028643 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:34.028665 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:34.028775 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:34.029121 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:34.029124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:34.029573 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:34.029730 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:34.029739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.030833 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.030837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:34.031759 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:34.031769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.032485 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.032489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:34.032612 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:34.033196 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:34.033213 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:34.035992 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":299,"threads_started":1}
W20260923 20:54:34.036473 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:34.028220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49931 (local address 127.0.97.62:35485)
0923 20:54:34.028412 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:34.028443 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:34.028727 (+ 284us) server_negotiation.cc:407] Connection header received
0923 20:54:34.028911 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:34.028925 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:34.028999 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:34.029080 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:34.029895 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.030697 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.031905 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.032339 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.032442 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:34.036232 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49931: BlockingRecv error: recv got EOF from 127.0.97.3:49931 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 20:54:34 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:34 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:34.101688 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:34.093239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:38127)
0923 20:54:34.093888 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:34.093918 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:34.094042 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:34.094293 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:34.094295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:34.094710 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:34.094881 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:34.094890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.095872 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.095875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:34.096796 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:34.096807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.097526 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.097530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:34.097640 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:34.098253 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:34.098271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:34.101339 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":258,"threads_started":1}
W20260923 20:54:34.101754 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:34.093290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38127 (local address 127.0.97.62:35485)
0923 20:54:34.093490 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:34.093498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:34.093999 (+ 501us) server_negotiation.cc:407] Connection header received
0923 20:54:34.094130 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:34.094134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:34.094198 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:34.094280 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:34.095014 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.095734 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.096970 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.097382 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.097500 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:34.101493 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38127: BlockingRecv error: recv got EOF from 127.0.97.1:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:54:34 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:34 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:34.576377 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:34.567389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:49535)
0923 20:54:34.568035 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:34.568055 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:34.568237 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:34.568625 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:34.568630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:34.569082 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:34.569341 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:34.569357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.570640 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.570645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:34.571649 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:34.571662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.572392 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.572398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:34.572592 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:34.573149 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:34.573166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:34.575996 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":277,"threads_started":1}
W20260923 20:54:34.576498 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:34.567438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49535 (local address 127.0.97.62:35485)
0923 20:54:34.567657 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:34.567683 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:34.568163 (+ 480us) server_negotiation.cc:407] Connection header received
0923 20:54:34.568404 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:34.568411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:34.568483 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:34.568591 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:34.569499 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.570497 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.571801 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:34.572235 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:34.572369 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:34.576244 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49535: BlockingRecv error: recv got EOF from 127.0.97.2:49535 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 23 20:54:35 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:35 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:35.051918 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:35.037274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:55951)
0923 20:54:35.038084 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:35.038113 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:35.038239 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:35.038539 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:35.038543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:35.038931 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:35.039113 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:35.039121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.040125 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.040129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:35.041058 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:35.041071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.041986 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.041995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:35.042192 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:35.048543 (+ 6351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:35.048565 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:35.051570 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":431,"threads_started":1}
W20260923 20:54:35.052030 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:35.037423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55951 (local address 127.0.97.62:35485)
0923 20:54:35.037599 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:35.037604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:35.038205 (+ 601us) server_negotiation.cc:407] Connection header received
0923 20:54:35.038344 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:35.038348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:35.038409 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:35.038494 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:35.039241 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.039967 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.041215 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.041702 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.041865 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:35.051804 (+ 9939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55951: BlockingRecv error: recv got EOF from 127.0.97.3:55951 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:54:35 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:35 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:35.111245 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:35.102780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:42213)
0923 20:54:35.103315 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:35.103345 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:35.103497 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:35.103918 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:35.103923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:35.104460 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:35.104699 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:35.104713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.105933 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.105937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:35.106843 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:35.106854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.107500 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.107505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:35.107626 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:35.108281 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:35.108298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:35.111010 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":310,"threads_started":1}
W20260923 20:54:35.111507 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:35.103006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42213 (local address 127.0.97.62:35485)
0923 20:54:35.103201 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:35.103205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:35.103440 (+ 235us) server_negotiation.cc:407] Connection header received
0923 20:54:35.103623 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:35.103651 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:35.103732 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:35.103859 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:35.104850 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.105762 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.106963 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.107351 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.107438 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:35.111211 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42213: BlockingRecv error: recv got EOF from 127.0.97.1:42213 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 20:54:35 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:35 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:35.586404 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:35.577451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:56709)
0923 20:54:35.578141 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:35.578167 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:35.578319 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:35.578763 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:35.578767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:35.579183 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:35.579339 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:35.579348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.580716 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.580721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:35.581666 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:35.581679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.582438 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.582443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:35.582575 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:35.583288 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:35.583311 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:35.586173 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":274,"threads_started":1}
W20260923 20:54:35.586733 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:35.577579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56709 (local address 127.0.97.62:35485)
0923 20:54:35.577911 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:35.577917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:35.578251 (+ 334us) server_negotiation.cc:407] Connection header received
0923 20:54:35.578532 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:35.578537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:35.578633 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:35.578727 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:35.579562 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.580537 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.581849 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:35.582260 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:35.582361 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:35.586417 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56709: BlockingRecv error: recv got EOF from 127.0.97.2:56709 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
Sep 23 20:54:36 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:36 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:36.062770 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:36.053221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:49275)
0923 20:54:36.054072 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:36.054093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:36.054269 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:36.054875 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:36.054880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:36.055368 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:36.055598 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:36.055608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.056708 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.056712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:36.057659 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:36.057671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.058527 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.058532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:36.058666 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:36.059398 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:36.059418 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:36.062539 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":609,"threads_started":1}
W20260923 20:54:36.063050 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:36.053332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49275 (local address 127.0.97.62:35485)
0923 20:54:36.053534 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:36.053540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:36.054190 (+ 650us) server_negotiation.cc:407] Connection header received
0923 20:54:36.054467 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:36.054473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:36.054571 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:36.054707 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:36.055803 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.056551 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.057875 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.058336 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.058453 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:36.062779 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49275: BlockingRecv error: recv got EOF from 127.0.97.3:49275 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:54:36 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:36 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:36.122253 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:36.112336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:52211)
0923 20:54:36.112959 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:36.112980 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:36.113113 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:36.113546 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:36.113550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:36.113961 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:36.114134 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:36.114144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.115329 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.115334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:36.116483 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:36.116500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.117617 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.117624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:36.117867 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:36.118631 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:36.118650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:36.121806 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":360,"threads_started":1}
W20260923 20:54:36.122392 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:36.112445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52211 (local address 127.0.97.62:35485)
0923 20:54:36.112691 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:36.112696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:36.113096 (+ 400us) server_negotiation.cc:407] Connection header received
0923 20:54:36.113223 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:36.113227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:36.113293 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:36.113392 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:36.114315 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.115113 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.116669 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.117447 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.117597 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:36.122184 (+ 4587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52211: BlockingRecv error: recv got EOF from 127.0.97.1:52211 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 23 20:54:36 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:36 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:36.596277 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:36.587402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:39623)
0923 20:54:36.587945 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:36.587967 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:36.588135 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:36.588474 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:36.588479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:36.588882 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:36.589041 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:36.589049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.590142 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.590149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:36.591342 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:36.591355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.592187 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.592196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:36.592397 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:36.593050 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:36.593069 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:36.596004 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":288,"threads_started":1}
W20260923 20:54:36.596432 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:36.587509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39623 (local address 127.0.97.62:35485)
0923 20:54:36.587765 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:36.587772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:36.588064 (+ 292us) server_negotiation.cc:407] Connection header received
0923 20:54:36.588278 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:36.588283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:36.588341 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:36.588448 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:36.589193 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.590013 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.591478 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:36.592020 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:36.592168 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:36.596200 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39623: BlockingRecv error: recv got EOF from 127.0.97.2:39623 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 20:54:37 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:37 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:37.072811 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:37.063939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:46351)
0923 20:54:37.064393 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:37.064414 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:37.064537 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:37.064891 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:37.064897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:37.065354 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:37.065518 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:37.065526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.066786 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.066793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:37.067829 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:37.067842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.068618 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.068624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:37.068827 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:37.069547 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:37.069565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:37.072560 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
W20260923 20:54:37.073045 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:37.064044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46351 (local address 127.0.97.62:35485)
0923 20:54:37.064285 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:37.064292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:37.064499 (+ 207us) server_negotiation.cc:407] Connection header received
0923 20:54:37.064661 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:37.064666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:37.064749 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:37.064831 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:37.065679 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.066643 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.068001 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.068456 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.068597 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:37.072820 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46351: BlockingRecv error: recv got EOF from 127.0.97.3:46351 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 23 20:54:37 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:37 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:37.131820 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:37.123227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:41623)
0923 20:54:37.123780 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:37.123802 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:37.123963 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:37.124341 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:37.124346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:37.124733 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:37.124881 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:37.124890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.125981 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.125988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:37.126930 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:37.126941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.127715 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.127721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:37.127904 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:37.128486 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:37.128504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:37.131468 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":296,"threads_started":1}
W20260923 20:54:37.131933 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:37.123390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41623 (local address 127.0.97.62:35485)
0923 20:54:37.123621 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:37.123628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:37.123890 (+ 262us) server_negotiation.cc:407] Connection header received
0923 20:54:37.124115 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:37.124120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:37.124192 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:37.124310 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:37.125037 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.125774 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.127080 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.127549 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.127688 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:37.131671 (+ 3983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41623: BlockingRecv error: recv got EOF from 127.0.97.1:41623 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 20:54:37 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:37 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:37.605896 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:37.597404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:34023)
0923 20:54:37.598013 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:37.598036 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:37.598193 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:37.598555 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:37.598559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:37.599131 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:37.599316 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:37.599325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.600428 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.600432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:37.601359 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:37.601370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.602135 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.602139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:37.602264 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:37.602890 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:37.602908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:37.605614 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":313,"threads_started":1}
W20260923 20:54:37.606127 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:37.597477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34023 (local address 127.0.97.62:35485)
0923 20:54:37.597705 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:37.597714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:37.598117 (+ 403us) server_negotiation.cc:407] Connection header received
0923 20:54:37.598324 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:37.598329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:37.598389 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:37.598516 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:37.599492 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.600297 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.601503 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:37.602000 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:37.602096 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:37.605894 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34023: BlockingRecv error: recv got EOF from 127.0.97.2:34023 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 20:54:38 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:38 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:38.083505 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:38.074011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:37297)
0923 20:54:38.074511 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:38.074533 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:38.074654 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:38.075094 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:38.075098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:38.075478 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:38.075631 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:38.075640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.076847 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.076852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:38.078027 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:38.078043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.078834 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.078840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:38.078976 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:38.079644 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:38.079664 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:38.083151 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
W20260923 20:54:38.083629 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:38.074223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37297 (local address 127.0.97.62:35485)
0923 20:54:38.074489 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:38.074497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:38.074637 (+ 140us) server_negotiation.cc:407] Connection header received
0923 20:54:38.074827 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:38.074831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:38.074910 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:38.074998 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:38.075795 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.076682 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.078191 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.078643 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.078765 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:38.083370 (+ 4605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37297: BlockingRecv error: recv got EOF from 127.0.97.3:37297 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 23 20:54:38 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:38 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:38.141188 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:38.132847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:45317)
0923 20:54:38.133402 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:38.133424 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:38.133543 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:38.133985 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:38.133988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:38.134418 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:38.134586 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:38.134595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.135653 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.135657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:38.136550 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:38.136561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.137246 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.137250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:38.137376 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:38.138128 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:38.138147 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:38.140943 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
W20260923 20:54:38.141423 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:38.132917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45317 (local address 127.0.97.62:35485)
0923 20:54:38.133100 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:38.133104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:38.133536 (+ 432us) server_negotiation.cc:407] Connection header received
0923 20:54:38.133670 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:38.133675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:38.133761 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:38.133956 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:38.134748 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.135532 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.136696 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.137099 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.137208 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:38.141177 (+ 3969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45317: BlockingRecv error: recv got EOF from 127.0.97.1:45317 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 20:54:38 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:38 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:38.615792 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:38.606961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:51617)
0923 20:54:38.607514 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:38.607535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:38.607704 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:38.608053 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:38.608058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:38.608495 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:38.608661 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:38.608669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.609700 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.609706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:38.610630 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:38.610642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.611366 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.611371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:38.611546 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:38.612219 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:38.612236 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:38.615554 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
W20260923 20:54:38.616021 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:38.607117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51617 (local address 127.0.97.62:35485)
0923 20:54:38.607343 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:38.607350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:38.607629 (+ 279us) server_negotiation.cc:407] Connection header received
0923 20:54:38.607851 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:38.607855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:38.607914 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:38.608026 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:38.608823 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.609559 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.610780 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:38.611197 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:38.611335 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:38.615742 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51617: BlockingRecv error: recv got EOF from 127.0.97.2:51617 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 20:54:39 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:39 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:39.093041 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:39.084561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:39993)
0923 20:54:39.085092 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:39.085112 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:39.085235 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:39.085530 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:39.085533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:39.086007 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:39.086190 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:39.086201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.087248 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.087253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:39.088176 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:39.088190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.088932 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.088937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:39.089055 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:39.089664 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:39.089680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:39.092670 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":297,"threads_started":1}
W20260923 20:54:39.093096 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:39.084815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39993 (local address 127.0.97.62:35485)
0923 20:54:39.084980 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:39.084984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:39.085189 (+ 205us) server_negotiation.cc:407] Connection header received
0923 20:54:39.085338 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:39.085341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:39.085402 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:39.085490 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:39.086350 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.087101 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.088343 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.088760 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.088855 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:39.092877 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39993: BlockingRecv error: recv got EOF from 127.0.97.3:39993 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 20:54:39 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:39 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:39.150627 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:39.142277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:49939)
0923 20:54:39.142852 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:39.142874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:39.142987 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:39.143299 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:39.143302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:39.143667 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:39.143819 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:39.143827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.144772 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.144776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:39.145685 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:39.145697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.146429 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.146433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:39.146565 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:39.147390 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:39.147408 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:39.150389 (+ 2981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":312,"threads_started":1}
W20260923 20:54:39.150848 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:39.142345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49939 (local address 127.0.97.62:35485)
0923 20:54:39.142525 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:39.142530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:39.142949 (+ 419us) server_negotiation.cc:407] Connection header received
0923 20:54:39.143082 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:39.143086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:39.143158 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:39.143242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:39.143939 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.144647 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.145824 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.146283 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.146388 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:39.150592 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49939: BlockingRecv error: recv got EOF from 127.0.97.1:49939 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 20:54:39 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:39 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:39.625684 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:39.616924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:33151)
0923 20:54:39.617485 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:39.617507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:39.617634 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:39.617983 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:39.617986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:39.618350 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:39.618509 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:39.618517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.619609 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.619614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:39.620992 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:39.621012 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.621897 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.621903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:39.622045 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:39.622718 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:39.622736 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:39.625449 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":312,"threads_started":1}
W20260923 20:54:39.625953 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:39.616990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33151 (local address 127.0.97.62:35485)
0923 20:54:39.617229 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:39.617240 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:39.617601 (+ 361us) server_negotiation.cc:407] Connection header received
0923 20:54:39.617742 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:39.617746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:39.617807 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:39.617917 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:39.618635 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.619448 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.621172 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:39.621697 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:39.621805 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:39.625641 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33151: BlockingRecv error: recv got EOF from 127.0.97.2:33151 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 20:54:40 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:40 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:40.106256 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:40.094016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:35441)
0923 20:54:40.094542 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:40.094575 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:40.094693 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:40.094993 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:40.094996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:40.095380 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:40.095556 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:40.095564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.096573 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.096577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:40.097532 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:40.097544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.098411 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.098416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:40.098547 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:40.103091 (+ 4544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:40.103122 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:40.106020 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":318,"threads_started":1}
W20260923 20:54:40.106525 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:40.094111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35441 (local address 127.0.97.62:35485)
0923 20:54:40.094278 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:40.094282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:40.094652 (+ 370us) server_negotiation.cc:407] Connection header received
0923 20:54:40.094798 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:40.094803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:40.094863 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:40.094948 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:40.095684 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.096413 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.097667 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.098272 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.098374 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:40.106237 (+ 7863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35441: BlockingRecv error: recv got EOF from 127.0.97.3:35441 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 20:54:40 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:40 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:40.160339 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:40.151742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:56619)
0923 20:54:40.152297 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:40.152317 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:40.152434 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:40.152739 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:40.152742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:40.153166 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:40.153336 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:40.153344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.154363 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.154367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:40.155282 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:40.155292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.156031 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.156035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:40.156150 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:40.156750 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:40.156768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:40.160077 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":302,"threads_started":1}
W20260923 20:54:40.160590 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:40.151780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56619 (local address 127.0.97.62:35485)
0923 20:54:40.151970 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:40.151977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:40.152393 (+ 416us) server_negotiation.cc:407] Connection header received
0923 20:54:40.152539 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:40.152543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:40.152618 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:40.152700 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:40.153478 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.154246 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.155426 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.155854 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.155981 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:40.160308 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56619: BlockingRecv error: recv got EOF from 127.0.97.1:56619 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 20:54:40 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:40 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:40.635603 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:40.626690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:59909)
0923 20:54:40.627230 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:40.627250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:40.627363 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:40.627667 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:40.627670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:40.628022 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:40.628196 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:40.628204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.629420 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.629425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:40.630832 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:40.630848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.631815 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.631820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:40.631938 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:40.632516 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:40.632533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:40.635352 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":272,"threads_started":1}
W20260923 20:54:40.635805 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:40.626826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59909 (local address 127.0.97.62:35485)
0923 20:54:40.626994 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:40.626998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:40.627319 (+ 321us) server_negotiation.cc:407] Connection header received
0923 20:54:40.627477 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:40.627481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:40.627540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:40.627630 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:40.628351 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.629257 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.630997 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:40.631646 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:40.631744 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:40.635574 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59909: BlockingRecv error: recv got EOF from 127.0.97.2:59909 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 20:54:41 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:41 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:41.115998 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:41.107343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:46479)
0923 20:54:41.107883 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:41.107905 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:41.108037 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:41.108348 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:41.108351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:41.108732 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:41.108913 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:41.108921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.110033 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.110037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:41.110991 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:41.111004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.111786 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.111794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:41.111969 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:41.112690 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:41.112736 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:41.115750 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
W20260923 20:54:41.116251 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:41.107409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46479 (local address 127.0.97.62:35485)
0923 20:54:41.107586 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:41.107591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:41.107984 (+ 393us) server_negotiation.cc:407] Connection header received
0923 20:54:41.108139 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:41.108143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:41.108214 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:41.108307 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:41.109059 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.109850 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.111153 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.111635 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.111739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:41.115979 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46479: BlockingRecv error: recv got EOF from 127.0.97.3:46479 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 20:54:41 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:41 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:41.169564 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:41.161468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:37227)
0923 20:54:41.162063 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:41.162092 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:41.162235 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:41.162562 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:41.162565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:41.162921 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:41.163070 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:41.163078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.164088 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.164093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:41.165019 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:41.165031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.165704 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.165708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:41.165857 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:41.166550 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:41.166568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:41.169303 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":350,"threads_started":1}
W20260923 20:54:41.169739 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:41.161497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37227 (local address 127.0.97.62:35485)
0923 20:54:41.161853 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:41.161862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:41.162184 (+ 322us) server_negotiation.cc:407] Connection header received
0923 20:54:41.162338 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:41.162342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:41.162434 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:41.162514 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:41.163195 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.163925 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.165160 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.165561 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.165654 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:41.169510 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37227: BlockingRecv error: recv got EOF from 127.0.97.1:37227 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Sep 23 20:54:41 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:41 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:41.645361 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:41.636668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:52377)
0923 20:54:41.637137 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:41.637159 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:41.637279 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:41.637574 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:41.637577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:41.638024 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:41.638181 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:41.638189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.639361 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.639364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:41.640445 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:41.640458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.641340 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.641345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:41.641492 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:41.642321 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:41.642339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:41.645105 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":267,"threads_started":1}
W20260923 20:54:41.645529 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:41.636744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52377 (local address 127.0.97.62:35485)
0923 20:54:41.636982 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:41.636988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:41.637243 (+ 255us) server_negotiation.cc:407] Connection header received
0923 20:54:41.637378 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:41.637382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:41.637443 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:41.637530 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:41.638413 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.639221 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.640586 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:41.641171 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:41.641274 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:41.645331 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52377: BlockingRecv error: recv got EOF from 127.0.97.2:52377 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 20:54:42 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:42 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:42.125589 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:42.117075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.3:34585)
0923 20:54:42.117607 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:42.117628 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:42.117755 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:42.118105 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:42.118108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:42.118475 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:42.118637 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:42.118646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.119816 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.119820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:42.120755 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:42.120766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.121415 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.121419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:42.121588 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.122268 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:42.122286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:42.125260 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":268,"threads_started":1}
W20260923 20:54:42.125837 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:42.117199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34585 (local address 127.0.97.62:35485)
0923 20:54:42.117490 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:42.117496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:42.117750 (+ 254us) server_negotiation.cc:407] Connection header received
0923 20:54:42.117890 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:42.117894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:42.117969 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:42.118067 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:42.118835 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.119653 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.120889 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.121282 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.121397 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.125498 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34585: BlockingRecv error: recv got EOF from 127.0.97.3:34585 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 23 20:54:42 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:42 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:42.179576 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:42.170982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.1:41177)
0923 20:54:42.171538 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:42.171559 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:42.171727 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:42.172102 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:42.172107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:42.172558 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:42.172722 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:42.172732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.173906 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.173919 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:42.174995 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:42.175011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.175764 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.175769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:42.175890 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.176504 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:42.176522 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:42.179310 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":316,"threads_started":1}
W20260923 20:54:42.179720 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:42.171112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41177 (local address 127.0.97.62:35485)
0923 20:54:42.171354 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:42.171363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:42.171655 (+ 292us) server_negotiation.cc:407] Connection header received
0923 20:54:42.171855 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:42.171861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:42.171947 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:42.172064 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:42.172883 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.173735 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.175155 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.175636 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.175731 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.179508 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41177: BlockingRecv error: recv got EOF from 127.0.97.1:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 20:54:42 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:54:42 dist-test-slave-plcg krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:54:42.655341 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:42.646637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35485 (local address 127.0.97.2:57925)
0923 20:54:42.647184 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:42.647203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:42.647317 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:42.647672 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:42.647694 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:42.648117 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:42.648286 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:42.648294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.649485 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.649489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:42.650426 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:42.650438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.651205 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.651209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:42.651326 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.651966 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:42.651984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:42.655030 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35485: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":322,"threads_started":1}
W20260923 20:54:42.655798 9403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:42.646719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57925 (local address 127.0.97.62:35485)
0923 20:54:42.646897 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:42.646902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:42.647280 (+ 378us) server_negotiation.cc:407] Connection header received
0923 20:54:42.647412 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:42.647416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:42.647475 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:42.647567 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:42.648457 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.649324 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.650563 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.651066 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.651178 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.655332 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57925: BlockingRecv error: recv got EOF from 127.0.97.2:57925 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:54:42.940675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9253
I20260923 20:54:42.953708 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:42.941870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.0.97.1:35229)
0923 20:54:42.942440 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:42.942448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:42.942475 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:54:42.942536 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:42.942542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:42.942771 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:42.942926 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:42.944505 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.945680 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.947101 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.947740 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.947846 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.950672 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:42.950680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:42.950694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:42.950734 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:42.953203 (+ 2469us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:54:42.953479 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54288: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":284,"threads_started":1}
W20260923 20:54:42.953860 9713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54288: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:54:42.954087 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9253: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35229: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:54:42.954309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9253
I20260923 20:54:42.971637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9404
I20260923 20:54:42.984246 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:42.972832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.97.2:46599)
0923 20:54:42.973544 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:42.973556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:42.973596 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:54:42.973658 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:42.973664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:42.973999 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:42.974163 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:42.975437 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.976553 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.977982 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:42.978573 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:42.978688 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:42.981202 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:42.981211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:42.981225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:42.981264 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:42.983703 (+ 2439us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:54:42.983967 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39060: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":336,"threads_started":1}
W20260923 20:54:42.984355 9714 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39060: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:54:42.984486 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9404: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:46599: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:54:42.984652 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9404
I20260923 20:54:42.998555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9535
I20260923 20:54:43.010638 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:42.999635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.0.97.3:46123)
0923 20:54:43.000168 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:43.000176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:43.000202 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:54:43.000262 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:43.000268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:43.000500 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:43.000656 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:43.002167 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:43.003190 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:43.004367 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:43.004901 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:43.004998 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:43.007522 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:43.007532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:43.007547 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:43.007593 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:43.010110 (+ 2517us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:54:43.010372 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59210: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":288,"threads_started":1}
W20260923 20:54:43.010743 9715 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59210: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:54:43.010807 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9535: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:46123: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:54:43.010993 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9535
I20260923 20:54:43.027958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9184
I20260923 20:54:43.029495 9246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:43.143891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9184
2026-09-23T20:54:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21315 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T20:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:43Z Disabled control of system clock
I20260923 20:54:43.222641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41319
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:54:43.557257 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:43.557646 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:43.557781 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:43.565788 9720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:43.566015 9720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:43.566092 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:43.566176 9720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:43.566264 9720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:43.576061 9720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:43.577524 9720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:43.579437 9720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:43.592509 9725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:43.592455 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:43.592484 9726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:43.592695 9720 server_base.cc:1034] running on GCE node
I20260923 20:54:43.593953 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:43.595516 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:43.596823 9720 hybrid_clock.cc:648] HybridClock initialized: now 1790196883596765 us; error 50 us; skew 500 ppm
I20260923 20:54:43.597280 9720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:43.600136 9720 webserver.cc:460] Webserver started at http://127.0.97.62:39357/ using document root <none> and password file <none>
I20260923 20:54:43.600951 9720 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:43.601095 9720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:43.601441 9720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:43.604697 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "faca393002ee489a88912ee90822cfdf"
format_stamp: "Formatted at 2026-09-23 20:54:43 on dist-test-slave-plcg"
server_key: "4745cdb34df9eec7b77b407201bca73a"
server_key_iv: "b092455049a71c4262aaf155dfe1c8f5"
server_key_version: "encryptionkey@0"
I20260923 20:54:43.605577 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "faca393002ee489a88912ee90822cfdf"
format_stamp: "Formatted at 2026-09-23 20:54:43 on dist-test-slave-plcg"
server_key: "4745cdb34df9eec7b77b407201bca73a"
server_key_iv: "b092455049a71c4262aaf155dfe1c8f5"
server_key_version: "encryptionkey@0"
I20260923 20:54:43.611275 9720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 20:54:43.615186 9734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:43.616711 9720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:54:43.616933 9720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "faca393002ee489a88912ee90822cfdf"
format_stamp: "Formatted at 2026-09-23 20:54:43 on dist-test-slave-plcg"
server_key: "4745cdb34df9eec7b77b407201bca73a"
server_key_iv: "b092455049a71c4262aaf155dfe1c8f5"
server_key_version: "encryptionkey@0"
I20260923 20:54:43.617187 9720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:43.631567 9720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:43.632493 9720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:43.632823 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:43.651916 9720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41319
I20260923 20:54:43.651923 9785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41319 every 8 connection(s)
I20260923 20:54:43.653368 9720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:43.655413 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9720
I20260923 20:54:43.655622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:54:43.656292 388 external_mini_cluster.cc:1428] Setting key 6d6fe79967d3c4ed9d516a582b968d10
I20260923 20:54:43.657869 9786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:43.671020 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:43.658575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.0.97.62:41319)
0923 20:54:43.659593 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:43.659604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:43.659648 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:54:43.661169 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:43.661192 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:43.661686 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:43.662132 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:43.662758 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:43.664105 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:43.666121 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:43.666836 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:43.666983 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:43.668452 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:43.668461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:43.668479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:43.669099 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:43.669194 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:43.669219 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:43.669318 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:43.669579 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 20:54:43.670023 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":386,"threads_started":1}
I20260923 20:54:43.672065 9786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf: Bootstrap starting.
I20260923 20:54:43.675305 9786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:43.676364 9786 log.cc:824] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:43.679621 9786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf: No bootstrap required, opened a new log
I20260923 20:54:43.687386 9786 raft_consensus.cc:348] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faca393002ee489a88912ee90822cfdf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41319 } }
I20260923 20:54:43.687824 9786 raft_consensus.cc:374] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:43.687942 9786 raft_consensus.cc:729] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faca393002ee489a88912ee90822cfdf, State: Initialized, Role: FOLLOWER
I20260923 20:54:43.688553 9786 consensus_queue.cc:260] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faca393002ee489a88912ee90822cfdf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41319 } }
I20260923 20:54:43.688802 9786 raft_consensus.cc:388] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:43.688956 9786 raft_consensus.cc:482] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:43.689154 9786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:43.691632 9786 raft_consensus.cc:504] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faca393002ee489a88912ee90822cfdf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41319 } }
I20260923 20:54:43.692203 9786 leader_election.cc:302] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faca393002ee489a88912ee90822cfdf; no voters:
I20260923 20:54:43.692975 9786 leader_election.cc:288] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:54:43.693182 9791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:43.694136 9791 raft_consensus.cc:686] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [term 1 LEADER]: Becoming Leader. State: Replica: faca393002ee489a88912ee90822cfdf, State: Running, Role: LEADER
I20260923 20:54:43.694686 9791 consensus_queue.cc:237] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faca393002ee489a88912ee90822cfdf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41319 } }
I20260923 20:54:43.695245 9786 sys_catalog.cc:564] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:43.697558 9792 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faca393002ee489a88912ee90822cfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faca393002ee489a88912ee90822cfdf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41319 } } }
I20260923 20:54:43.697767 9793 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader faca393002ee489a88912ee90822cfdf. Latest consensus state: current_term: 1 leader_uuid: "faca393002ee489a88912ee90822cfdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faca393002ee489a88912ee90822cfdf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41319 } } }
I20260923 20:54:43.698071 9792 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:43.698079 9793 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:43.699045 9799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:43.705552 9799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:43.715556 9799 catalog_manager.cc:1347] Generated new cluster ID: 9bf9b7edf93641f5bc74cba0138d57f3
I20260923 20:54:43.715724 9799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:43.760466 9799 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:54:43.761899 9799 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:43.769718 9799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faca393002ee489a88912ee90822cfdf: Generated new TSK 0
I20260923 20:54:43.770874 9799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:54:43.789474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41319
--builtin_ntp_servers=127.0.97.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 20:54:44.159209 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:44.159514 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:44.159596 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:44.166966 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:44.167088 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:44.167191 9810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:44.176692 9810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:44.178166 9810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:44.179913 9810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:44.193867 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:44.193867 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:44.194196 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:44.194643 9810 server_base.cc:1034] running on GCE node
I20260923 20:54:44.195348 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:44.196523 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:44.197817 9810 hybrid_clock.cc:648] HybridClock initialized: now 1790196884197735 us; error 60 us; skew 500 ppm
I20260923 20:54:44.198168 9810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:44.200529 9810 webserver.cc:460] Webserver started at http://127.0.97.1:34353/ using document root <none> and password file <none>
I20260923 20:54:44.201287 9810 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:44.201380 9810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:44.201663 9810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:44.204944 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "46cfd78725b44937bc03017c68284bb6"
format_stamp: "Formatted at 2026-09-23 20:54:44 on dist-test-slave-plcg"
server_key: "e62e5ba01dc3f9be356a7be82dcf69d6"
server_key_iv: "31ec981f3eb7694c44f04165a9102919"
server_key_version: "encryptionkey@0"
I20260923 20:54:44.205797 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "46cfd78725b44937bc03017c68284bb6"
format_stamp: "Formatted at 2026-09-23 20:54:44 on dist-test-slave-plcg"
server_key: "e62e5ba01dc3f9be356a7be82dcf69d6"
server_key_iv: "31ec981f3eb7694c44f04165a9102919"
server_key_version: "encryptionkey@0"
I20260923 20:54:44.211283 9810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:54:44.214751 9824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:44.216132 9810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:54:44.216281 9810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "46cfd78725b44937bc03017c68284bb6"
format_stamp: "Formatted at 2026-09-23 20:54:44 on dist-test-slave-plcg"
server_key: "e62e5ba01dc3f9be356a7be82dcf69d6"
server_key_iv: "31ec981f3eb7694c44f04165a9102919"
server_key_version: "encryptionkey@0"
I20260923 20:54:44.216470 9810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:44.230285 9810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:44.231096 9810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:44.231331 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:44.232249 9810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:44.233709 9810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:44.233798 9810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:44.233955 9810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:44.234018 9810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:44.267632 9810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36779
I20260923 20:54:44.267675 9936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36779 every 8 connection(s)
I20260923 20:54:44.269183 9810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:44.269572 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9810
I20260923 20:54:44.269785 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:54:44.270648 388 external_mini_cluster.cc:1428] Setting key cc04718a37e9d3941f4051c207e543fc
I20260923 20:54:44.274117 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41319
--builtin_ntp_servers=127.0.97.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:54:44.283013 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:44.272295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35575 (local address 127.0.97.62:41319)
0923 20:54:44.272740 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:44.272746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:44.276232 (+ 3486us) server_negotiation.cc:407] Connection header received
0923 20:54:44.276292 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:44.276296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:44.276377 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:44.276507 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:44.277658 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.278444 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.280084 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.280503 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.280646 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:44.282153 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:44.282158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:44.282162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:44.282182 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:44.282219 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:44.282231 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:44.282329 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:44.282569 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 20:54:44.282671 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260923 20:54:44.284139 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:44.273083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41319 (local address 127.0.97.1:35575)
0923 20:54:44.274220 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:44.274281 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:44.275920 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:44.276767 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:44.276776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:44.276874 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:44.277473 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:44.277492 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.278613 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.278620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:44.279901 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:44.279914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.280747 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.280754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:44.281004 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:44.281742 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:54:44.281744 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:44.281789 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:54:44.281928 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:44.282327 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:44.282339 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:44.282350 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:54:44.282559 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 20:54:44.283005 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":339,"threads_started":1}
I20260923 20:54:44.285859 9937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41319
I20260923 20:54:44.286177 9937 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:44.286926 9937 heartbeater.cc:511] Master 127.0.97.62:41319 requested a full tablet report, sending...
I20260923 20:54:44.289284 9751 ts_manager.cc:194] Registered new tserver with Master: 46cfd78725b44937bc03017c68284bb6 (127.0.97.1:36779)
I20260923 20:54:44.291740 9751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35575
W20260923 20:54:44.610867 9941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:44.611186 9941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:44.611274 9941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:44.618798 9941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:44.618927 9941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:44.619036 9941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:44.628701 9941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:44.630033 9941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:44.631747 9941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:44.645146 9947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:44.645187 9949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:44.645064 9946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:44.646184 9941 server_base.cc:1034] running on GCE node
I20260923 20:54:44.646883 9941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:44.647962 9941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:44.649207 9941 hybrid_clock.cc:648] HybridClock initialized: now 1790196884649157 us; error 52 us; skew 500 ppm
I20260923 20:54:44.649569 9941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:44.652673 9941 webserver.cc:460] Webserver started at http://127.0.97.2:39117/ using document root <none> and password file <none>
I20260923 20:54:44.653612 9941 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:44.653745 9941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:44.654170 9941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:44.658224 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "8bc1fb4dcbaa4fc5bc55fbfb0b51da79"
format_stamp: "Formatted at 2026-09-23 20:54:44 on dist-test-slave-plcg"
server_key: "aac1863693346e61e0d9b6b8ae18fef0"
server_key_iv: "e599f76addc449dc0d4776b713d0aa03"
server_key_version: "encryptionkey@0"
I20260923 20:54:44.659260 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8bc1fb4dcbaa4fc5bc55fbfb0b51da79"
format_stamp: "Formatted at 2026-09-23 20:54:44 on dist-test-slave-plcg"
server_key: "aac1863693346e61e0d9b6b8ae18fef0"
server_key_iv: "e599f76addc449dc0d4776b713d0aa03"
server_key_version: "encryptionkey@0"
I20260923 20:54:44.666057 9941 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:54:44.670786 9955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:44.672518 9941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:54:44.672704 9941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "8bc1fb4dcbaa4fc5bc55fbfb0b51da79"
format_stamp: "Formatted at 2026-09-23 20:54:44 on dist-test-slave-plcg"
server_key: "aac1863693346e61e0d9b6b8ae18fef0"
server_key_iv: "e599f76addc449dc0d4776b713d0aa03"
server_key_version: "encryptionkey@0"
I20260923 20:54:44.672966 9941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:44.704305 9941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:44.705179 9941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:44.705425 9941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:44.706516 9941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:44.708081 9941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:44.708174 9941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:44.708288 9941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:44.708362 9941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:44.742923 9941 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36029
I20260923 20:54:44.742985 10067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36029 every 8 connection(s)
I20260923 20:54:44.744261 9941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:44.747813 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 9941
I20260923 20:54:44.747995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:54:44.748445 388 external_mini_cluster.cc:1428] Setting key 80ebac1cb91e444bcaf39c928432d4da
I20260923 20:54:44.751516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41319
--builtin_ntp_servers=127.0.97.20:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:54:44.758428 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:44.746882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49135 (local address 127.0.97.62:41319)
0923 20:54:44.747055 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:44.747060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:44.748741 (+ 1681us) server_negotiation.cc:407] Connection header received
0923 20:54:44.751025 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:44.751030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:44.751112 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:44.751228 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:44.752368 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.753571 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.755200 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.755957 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.756104 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:44.757483 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:44.757492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:44.757500 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:44.757528 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:44.757570 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:44.757588 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:44.757695 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:44.757939 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:54:44.758056 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:54:44.759377 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:44.747392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41319 (local address 127.0.97.2:49135)
0923 20:54:44.748540 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:44.748585 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:44.750735 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:44.751429 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:44.751438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:44.751530 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:44.752192 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:44.752209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.753714 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.753717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:44.755040 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:44.755052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:44.756168 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:44.756173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:44.756374 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:44.757146 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:54:44.757148 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:44.757194 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:54:44.757276 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:44.757692 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:44.757699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:44.757705 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:54:44.757917 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 20:54:44.758230 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":361,"threads_started":1}
I20260923 20:54:44.760869 10068 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41319
I20260923 20:54:44.761196 10068 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:44.762068 10068 heartbeater.cc:511] Master 127.0.97.62:41319 requested a full tablet report, sending...
I20260923 20:54:44.763602 9751 ts_manager.cc:194] Registered new tserver with Master: 8bc1fb4dcbaa4fc5bc55fbfb0b51da79 (127.0.97.2:36029)
I20260923 20:54:44.764444 9751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49135
W20260923 20:54:45.097770 10071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:45.098143 10071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:45.098309 10071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:45.105921 10071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:45.106119 10071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:45.106376 10071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:45.116094 10071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:45.117426 10071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:45.119351 10071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:45.133191 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:45.133255 10077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:45.133512 10076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:45.134284 10071 server_base.cc:1034] running on GCE node
I20260923 20:54:45.134943 10071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:45.135972 10071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:45.137226 10071 hybrid_clock.cc:648] HybridClock initialized: now 1790196885137177 us; error 46 us; skew 500 ppm
I20260923 20:54:45.137583 10071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:45.145486 10071 webserver.cc:460] Webserver started at http://127.0.97.3:45081/ using document root <none> and password file <none>
I20260923 20:54:45.146534 10071 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:45.146675 10071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:45.147042 10071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:45.150153 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "e29805e3d20b424eb53fc9a162286d91"
format_stamp: "Formatted at 2026-09-23 20:54:45 on dist-test-slave-plcg"
server_key: "97003722d94ff1f4432627315a28be36"
server_key_iv: "bdf0614bcb84473ae967f6d2c99858c1"
server_key_version: "encryptionkey@0"
I20260923 20:54:45.151046 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e29805e3d20b424eb53fc9a162286d91"
format_stamp: "Formatted at 2026-09-23 20:54:45 on dist-test-slave-plcg"
server_key: "97003722d94ff1f4432627315a28be36"
server_key_iv: "bdf0614bcb84473ae967f6d2c99858c1"
server_key_version: "encryptionkey@0"
I20260923 20:54:45.156769 10071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:54:45.160744 10085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:45.162477 10071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:45.162701 10071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "e29805e3d20b424eb53fc9a162286d91"
format_stamp: "Formatted at 2026-09-23 20:54:45 on dist-test-slave-plcg"
server_key: "97003722d94ff1f4432627315a28be36"
server_key_iv: "bdf0614bcb84473ae967f6d2c99858c1"
server_key_version: "encryptionkey@0"
I20260923 20:54:45.162948 10071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:45.182122 10071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:45.183243 10071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:45.183568 10071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:45.184602 10071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:45.186241 10071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:45.186371 10071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:45.186507 10071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:45.186595 10071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:45.222609 10071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45685
I20260923 20:54:45.222661 10197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45685 every 8 connection(s)
I20260923 20:54:45.224080 10071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:45.227121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10071
I20260923 20:54:45.227308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:54:45.227790 388 external_mini_cluster.cc:1428] Setting key bd2a1d08f365dbde690c0d1b7002941c
I20260923 20:54:45.238022 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:45.226887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60507 (local address 127.0.97.62:41319)
0923 20:54:45.227149 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:45.227158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:45.228991 (+ 1833us) server_negotiation.cc:407] Connection header received
0923 20:54:45.230968 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:45.230974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:45.231074 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:45.231202 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:45.232374 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.233216 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.235056 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.235487 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.235636 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:45.237131 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:45.237136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:45.237139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:45.237158 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:45.237191 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:45.237203 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:45.237306 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:45.237541 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 20:54:45.237643 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:54:45.239151 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:45.227601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41319 (local address 127.0.97.3:60507)
0923 20:54:45.228798 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:45.228853 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:45.230678 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:45.231462 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:45.231471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:45.231578 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:45.232192 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:45.232210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.233398 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.233403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:45.234877 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:45.234895 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.235759 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.235767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:45.236029 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:45.236766 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:54:45.236769 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:45.236811 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:54:45.236895 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:45.237304 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:45.237316 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:45.237327 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:54:45.237530 (+ 203us) client_negotiation.cc:240] Negotiation successful
0923 20:54:45.238004 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":403,"threads_started":1}
I20260923 20:54:45.240754 10198 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41319
I20260923 20:54:45.241147 10198 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:45.242241 10198 heartbeater.cc:511] Master 127.0.97.62:41319 requested a full tablet report, sending...
I20260923 20:54:45.243768 9751 ts_manager.cc:194] Registered new tserver with Master: e29805e3d20b424eb53fc9a162286d91 (127.0.97.3:45685)
I20260923 20:54:45.244562 9751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60507
I20260923 20:54:45.255621 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:54:45.262159 9939 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:45.260529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.97.62:41319)
0923 20:54:45.260779 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:45.260783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:45.261202 (+ 419us) server_negotiation.cc:407] Connection header received
0923 20:54:45.261347 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:45.261351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:45.261426 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:45.261522 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:45.261972 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58228: BlockingRecv error: recv got EOF from 127.0.0.1:58228 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:54:45.262861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9810
I20260923 20:54:45.271864 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:45.263818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.0.97.1:36779)
0923 20:54:45.264357 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:45.264364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:45.264392 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:54:45.264450 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:45.264457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:45.264675 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:45.264806 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:45.265169 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.266227 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.267815 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.268766 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.268906 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:45.270049 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:45.270060 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:45.270076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:45.270823 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:45.270887 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:45.270913 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:45.271045 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:45.271338 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 20:54:45.271475 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":296,"threads_started":1}
I20260923 20:54:45.273221 9932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:45.355172 9937 heartbeater.cc:503] Master 127.0.97.62:41319 was elected leader, sending a full tablet report...
I20260923 20:54:45.356215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9810
W20260923 20:54:45.374033 9735 connection.cc:535] server connection from 127.0.97.1:35575 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:54:45.374464 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9941
I20260923 20:54:45.383118 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:45.375373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.0.97.2:36029)
0923 20:54:45.375865 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:45.375872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:45.375899 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:54:45.375961 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:45.375967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:45.376255 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:45.376432 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:45.376846 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.377959 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.379532 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.380274 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.380420 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:45.381560 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:45.381575 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:45.381589 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:45.382277 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:45.382338 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:45.382357 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:45.382487 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:45.382626 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 20:54:45.382770 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260923 20:54:45.384528 10063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:45.467279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9941
I20260923 20:54:45.483856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10071
I20260923 20:54:45.492604 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:45.484770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.0.97.3:45685)
0923 20:54:45.485403 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:45.485411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:45.485441 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:54:45.485501 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:45.485507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:45.485733 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:45.485985 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:45.486447 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.487625 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.489007 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:45.489614 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:45.489772 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:45.491089 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:45.491097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:45.491113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:45.491731 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:45.491781 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:45.491807 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:45.491939 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:45.492094 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 20:54:45.492253 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":305,"threads_started":1}
I20260923 20:54:45.494217 10193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:45.577135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10071
I20260923 20:54:45.593248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 9720
I20260923 20:54:45.594715 9781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:45.699262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 9720
2026-09-23T20:54:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2545 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T20:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:45Z Disabled control of system clock
I20260923 20:54:45.769074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46323
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:54:46.117041 10217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:46.117347 10217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:46.117434 10217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:46.124842 10217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:46.124961 10217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:46.125001 10217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:46.125052 10217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:46.125097 10217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:46.134524 10217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:46.135818 10217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:46.137723 10217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:46.149891 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:46.149927 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:46.149974 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:46.150857 10217 server_base.cc:1034] running on GCE node
I20260923 20:54:46.151739 10217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:46.153275 10217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:46.154539 10217 hybrid_clock.cc:648] HybridClock initialized: now 1790196886154485 us; error 48 us; skew 500 ppm
I20260923 20:54:46.154969 10217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:46.160953 10217 webserver.cc:460] Webserver started at http://127.0.97.62:43289/ using document root <none> and password file <none>
I20260923 20:54:46.161751 10217 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:46.161937 10217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:46.162289 10217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:46.165364 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "8092f7eac3d543f7a0bbfee29703d65a"
format_stamp: "Formatted at 2026-09-23 20:54:46 on dist-test-slave-plcg"
server_key: "c6e7e60915959fb70bab940790302ec7"
server_key_iv: "2f0e6d29441024b225e7e9d5f20b96ed"
server_key_version: "encryptionkey@0"
I20260923 20:54:46.166280 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "8092f7eac3d543f7a0bbfee29703d65a"
format_stamp: "Formatted at 2026-09-23 20:54:46 on dist-test-slave-plcg"
server_key: "c6e7e60915959fb70bab940790302ec7"
server_key_iv: "2f0e6d29441024b225e7e9d5f20b96ed"
server_key_version: "encryptionkey@0"
I20260923 20:54:46.171736 10217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:54:46.175820 10231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:46.177279 10217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:54:46.177471 10217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "8092f7eac3d543f7a0bbfee29703d65a"
format_stamp: "Formatted at 2026-09-23 20:54:46 on dist-test-slave-plcg"
server_key: "c6e7e60915959fb70bab940790302ec7"
server_key_iv: "2f0e6d29441024b225e7e9d5f20b96ed"
server_key_version: "encryptionkey@0"
I20260923 20:54:46.177712 10217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:46.183710 10217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:46.184564 10217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:46.184876 10217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:46.203775 10217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46323
I20260923 20:54:46.203809 10282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46323 every 8 connection(s)
I20260923 20:54:46.205201 10217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:46.208997 10283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:46.212550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10217
I20260923 20:54:46.212782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:54:46.213577 388 external_mini_cluster.cc:1428] Setting key eccdcc233fbfb59d2181be2dba1a04ed
I20260923 20:54:46.222554 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:46.215873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.0.97.62:46323)
0923 20:54:46.216884 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:46.216896 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:46.216937 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:54:46.218693 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:46.218717 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:46.219204 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:46.219634 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:46.220784 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:46.220790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:46.220804 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:46.220818 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:46.220878 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:46.220893 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:46.221142 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:46.221279 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 20:54:46.221534 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":366,"threads_started":1}
I20260923 20:54:46.222549 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a: Bootstrap starting.
I20260923 20:54:46.225888 10283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:46.227288 10283 log.cc:824] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:46.230113 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a: No bootstrap required, opened a new log
I20260923 20:54:46.237608 10283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8092f7eac3d543f7a0bbfee29703d65a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46323 } }
I20260923 20:54:46.238125 10283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:46.238237 10283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8092f7eac3d543f7a0bbfee29703d65a, State: Initialized, Role: FOLLOWER
I20260923 20:54:46.238904 10283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8092f7eac3d543f7a0bbfee29703d65a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46323 } }
I20260923 20:54:46.239168 10283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:46.239316 10283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:46.239497 10283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:46.242110 10283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8092f7eac3d543f7a0bbfee29703d65a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46323 } }
I20260923 20:54:46.242657 10283 leader_election.cc:302] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8092f7eac3d543f7a0bbfee29703d65a; no voters:
I20260923 20:54:46.243286 10283 leader_election.cc:288] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:54:46.243453 10288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:46.244273 10288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [term 1 LEADER]: Becoming Leader. State: Replica: 8092f7eac3d543f7a0bbfee29703d65a, State: Running, Role: LEADER
I20260923 20:54:46.244952 10288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8092f7eac3d543f7a0bbfee29703d65a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46323 } }
I20260923 20:54:46.245549 10283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:46.248005 10290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8092f7eac3d543f7a0bbfee29703d65a. Latest consensus state: current_term: 1 leader_uuid: "8092f7eac3d543f7a0bbfee29703d65a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8092f7eac3d543f7a0bbfee29703d65a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46323 } } }
I20260923 20:54:46.248500 10290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:46.248003 10289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8092f7eac3d543f7a0bbfee29703d65a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8092f7eac3d543f7a0bbfee29703d65a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46323 } } }
I20260923 20:54:46.248769 10289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:46.249392 10295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:46.254168 10295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:46.264392 10295 catalog_manager.cc:1347] Generated new cluster ID: 86d7097269684b46bd18b07651b2cd90
I20260923 20:54:46.264954 10295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:46.280328 10295 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:54:46.280797 10295 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:46.292520 10295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8092f7eac3d543f7a0bbfee29703d65a: Generated new TSK 0
I20260923 20:54:46.293474 10295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:54:46.304658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46323
--builtin_ntp_servers=127.0.97.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:54:46.699296 10307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:46.699605 10307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:46.699707 10307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:46.707177 10307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:46.707293 10307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:46.707405 10307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:46.716883 10307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:46.718180 10307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:46.719985 10307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:46.733177 10312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:46.733160 10315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:46.733199 10313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:46.734472 10307 server_base.cc:1034] running on GCE node
I20260923 20:54:46.735124 10307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:46.736158 10307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:46.737397 10307 hybrid_clock.cc:648] HybridClock initialized: now 1790196886737342 us; error 71 us; skew 500 ppm
I20260923 20:54:46.737730 10307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:46.740265 10307 webserver.cc:460] Webserver started at http://127.0.97.1:41585/ using document root <none> and password file <none>
I20260923 20:54:46.741015 10307 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:46.741307 10307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:46.741667 10307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:46.744781 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "70ae9fc70ed245f4afcad1f09b8322c9"
format_stamp: "Formatted at 2026-09-23 20:54:46 on dist-test-slave-plcg"
server_key: "e624ca4394b3a09fb9578711aae71007"
server_key_iv: "5460e1d00c6123fe34b168aaf511ab72"
server_key_version: "encryptionkey@0"
I20260923 20:54:46.745702 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "70ae9fc70ed245f4afcad1f09b8322c9"
format_stamp: "Formatted at 2026-09-23 20:54:46 on dist-test-slave-plcg"
server_key: "e624ca4394b3a09fb9578711aae71007"
server_key_iv: "5460e1d00c6123fe34b168aaf511ab72"
server_key_version: "encryptionkey@0"
I20260923 20:54:46.751715 10307 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:54:46.755419 10321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:46.756835 10307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:54:46.757017 10307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "70ae9fc70ed245f4afcad1f09b8322c9"
format_stamp: "Formatted at 2026-09-23 20:54:46 on dist-test-slave-plcg"
server_key: "e624ca4394b3a09fb9578711aae71007"
server_key_iv: "5460e1d00c6123fe34b168aaf511ab72"
server_key_version: "encryptionkey@0"
I20260923 20:54:46.757274 10307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:46.763156 10307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:46.764091 10307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:46.764406 10307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:46.765398 10307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:46.767040 10307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:46.767177 10307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:46.767316 10307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:46.767405 10307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:46.800508 10307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34369
I20260923 20:54:46.800552 10433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34369 every 8 connection(s)
I20260923 20:54:46.802018 10307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:46.809113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10307
I20260923 20:54:46.809401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:54:46.810577 388 external_mini_cluster.cc:1428] Setting key cc0ee069be998ab5937dad3b80cd3a2d
I20260923 20:54:46.811719 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:46.804824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55131 (local address 127.0.97.62:46323)
0923 20:54:46.805338 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:46.805342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:46.806628 (+ 1286us) server_negotiation.cc:407] Connection header received
0923 20:54:46.808410 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:46.808416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:46.808490 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:46.808632 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:46.810778 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:46.810784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:46.810789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:46.810816 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:46.810885 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:46.810904 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:46.811054 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:46.811303 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 20:54:46.811376 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":260,"threads_started":1}
I20260923 20:54:46.812798 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:46.805417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46323 (local address 127.0.97.1:55131)
0923 20:54:46.806458 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:46.806495 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:46.808093 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:46.808909 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:46.808918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:46.809024 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:46.810029 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:54:46.810036 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:46.810125 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:54:46.810339 (+ 214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:46.811016 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:46.811048 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:46.811057 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:54:46.811396 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 20:54:46.811723 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":389,"threads_started":1}
I20260923 20:54:46.813657 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46323
--builtin_ntp_servers=127.0.97.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 20:54:46.814544 10434 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46323
I20260923 20:54:46.814884 10434 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:46.815635 10434 heartbeater.cc:511] Master 127.0.97.62:46323 requested a full tablet report, sending...
I20260923 20:54:46.818120 10248 ts_manager.cc:194] Registered new tserver with Master: 70ae9fc70ed245f4afcad1f09b8322c9 (127.0.97.1:34369)
W20260923 20:54:47.151603 10438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:47.151908 10438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:47.152052 10438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:47.159770 10438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:47.159890 10438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:47.160068 10438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:47.169649 10438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:47.171013 10438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:47.172837 10438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:47.186793 10446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:47.186880 10443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:47.186880 10444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:47.188289 10438 server_base.cc:1034] running on GCE node
I20260923 20:54:47.188930 10438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:47.189940 10438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:47.191149 10438 hybrid_clock.cc:648] HybridClock initialized: now 1790196887191104 us; error 50 us; skew 500 ppm
I20260923 20:54:47.191588 10438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:47.194012 10438 webserver.cc:460] Webserver started at http://127.0.97.2:44651/ using document root <none> and password file <none>
I20260923 20:54:47.194792 10438 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:47.194883 10438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:47.195148 10438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:47.198247 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "672075512fe247eea3b6f219eb57bb19"
format_stamp: "Formatted at 2026-09-23 20:54:47 on dist-test-slave-plcg"
server_key: "914a3fd047b640c33caae7b26252e425"
server_key_iv: "76f209044540923a41013234a38e464a"
server_key_version: "encryptionkey@0"
I20260923 20:54:47.199054 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "672075512fe247eea3b6f219eb57bb19"
format_stamp: "Formatted at 2026-09-23 20:54:47 on dist-test-slave-plcg"
server_key: "914a3fd047b640c33caae7b26252e425"
server_key_iv: "76f209044540923a41013234a38e464a"
server_key_version: "encryptionkey@0"
I20260923 20:54:47.204738 10438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:54:47.209498 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:47.211277 10438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:54:47.211459 10438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "672075512fe247eea3b6f219eb57bb19"
format_stamp: "Formatted at 2026-09-23 20:54:47 on dist-test-slave-plcg"
server_key: "914a3fd047b640c33caae7b26252e425"
server_key_iv: "76f209044540923a41013234a38e464a"
server_key_version: "encryptionkey@0"
I20260923 20:54:47.211743 10438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:47.219376 10438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:47.220400 10438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:47.220713 10438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:47.221942 10438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:47.223773 10438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:47.223892 10438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:47.224009 10438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:47.224078 10438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:47.258533 10438 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45395
I20260923 20:54:47.258587 10564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45395 every 8 connection(s)
I20260923 20:54:47.259939 10438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:47.267885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10438
I20260923 20:54:47.268079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:54:47.268590 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:47.262803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32887 (local address 127.0.97.62:46323)
0923 20:54:47.263001 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:47.263005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:47.264404 (+ 1399us) server_negotiation.cc:407] Connection header received
0923 20:54:47.266202 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:47.266207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:47.266284 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:47.266396 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:47.267890 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:47.267894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:47.267898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:47.267915 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:47.267946 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:47.267981 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:47.268145 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:47.268344 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 20:54:47.268397 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:54:47.268612 388 external_mini_cluster.cc:1428] Setting key bb6015fa6d9c6ae91680cd984878ce0f
I20260923 20:54:47.269908 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:47.263240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46323 (local address 127.0.97.2:32887)
0923 20:54:47.264236 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:47.264279 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:47.265902 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:47.266639 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:47.266647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:47.266745 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:47.267490 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:54:47.267493 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:47.267549 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:54:47.267674 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:47.268112 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:47.268123 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:47.268132 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:54:47.268420 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 20:54:47.268701 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":298,"threads_started":1}
I20260923 20:54:47.271394 10565 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46323
I20260923 20:54:47.271596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46323
--builtin_ntp_servers=127.0.97.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 20:54:47.271787 10565 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:47.272403 10565 heartbeater.cc:511] Master 127.0.97.62:46323 requested a full tablet report, sending...
I20260923 20:54:47.273970 10248 ts_manager.cc:194] Registered new tserver with Master: 672075512fe247eea3b6f219eb57bb19 (127.0.97.2:45395)
W20260923 20:54:47.612352 10568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:47.612718 10568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:47.612888 10568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:47.620761 10568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:47.620895 10568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:47.621009 10568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:47.630722 10568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:47.632025 10568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:47.633926 10568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:47.647085 10573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:47.647714 10574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:47.648206 10576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:47.648928 10568 server_base.cc:1034] running on GCE node
I20260923 20:54:47.649650 10568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:47.650729 10568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:47.652009 10568 hybrid_clock.cc:648] HybridClock initialized: now 1790196887651935 us; error 66 us; skew 500 ppm
I20260923 20:54:47.652383 10568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:54:47.655005 10568 webserver.cc:460] Webserver started at http://127.0.97.3:40379/ using document root <none> and password file <none>
I20260923 20:54:47.655781 10568 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:47.655936 10568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:47.656268 10568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:47.659351 10568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "a87d933169b8497b94b4491c00a800a3"
format_stamp: "Formatted at 2026-09-23 20:54:47 on dist-test-slave-plcg"
server_key: "e0fb3f597d8aeaf4063241af76980eb5"
server_key_iv: "71c79dd1e9bd484a71251fd20350282c"
server_key_version: "encryptionkey@0"
I20260923 20:54:47.660200 10568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a87d933169b8497b94b4491c00a800a3"
format_stamp: "Formatted at 2026-09-23 20:54:47 on dist-test-slave-plcg"
server_key: "e0fb3f597d8aeaf4063241af76980eb5"
server_key_iv: "71c79dd1e9bd484a71251fd20350282c"
server_key_version: "encryptionkey@0"
I20260923 20:54:47.665781 10568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:54:47.669459 10582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:47.670979 10568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:47.671182 10568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "a87d933169b8497b94b4491c00a800a3"
format_stamp: "Formatted at 2026-09-23 20:54:47 on dist-test-slave-plcg"
server_key: "e0fb3f597d8aeaf4063241af76980eb5"
server_key_iv: "71c79dd1e9bd484a71251fd20350282c"
server_key_version: "encryptionkey@0"
I20260923 20:54:47.671438 10568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:47.677695 10568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:47.678625 10568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:47.678922 10568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:47.679898 10568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:47.681538 10568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:47.681666 10568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:47.681849 10568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:47.681953 10568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 20:54:47.716208 10568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32913
I20260923 20:54:47.716320 10694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32913 every 8 connection(s)
I20260923 20:54:47.717656 10568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:47.725571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10568
I20260923 20:54:47.725842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:54:47.725875 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:47.720258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49199 (local address 127.0.97.62:46323)
0923 20:54:47.720479 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:47.720485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:47.721989 (+ 1504us) server_negotiation.cc:407] Connection header received
0923 20:54:47.723619 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:47.723623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:47.723677 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:47.723766 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:47.725106 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:47.725109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:47.725113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:47.725126 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:47.725150 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:47.725160 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:47.725261 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:47.725518 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 20:54:47.725567 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:54:47.726383 388 external_mini_cluster.cc:1428] Setting key cad1157357a0c0de2c186b855cb2249f
I20260923 20:54:47.727257 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:47.720816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46323 (local address 127.0.97.3:49199)
0923 20:54:47.721762 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:47.721805 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:47.723364 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:47.723984 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:47.723993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:47.724088 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:47.724821 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:54:47.724824 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:47.724863 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:54:47.724952 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:47.725279 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:47.725290 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:47.725301 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:54:47.725623 (+ 322us) client_negotiation.cc:240] Negotiation successful
0923 20:54:47.725958 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":276,"threads_started":1}
I20260923 20:54:47.728575 10695 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46323
I20260923 20:54:47.728914 10695 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:47.729610 10695 heartbeater.cc:511] Master 127.0.97.62:46323 requested a full tablet report, sending...
I20260923 20:54:47.731055 10248 ts_manager.cc:194] Registered new tserver with Master: a87d933169b8497b94b4491c00a800a3 (127.0.97.3:32913)
I20260923 20:54:47.741966 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:54:47.748565 10436 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:54:47.746795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.0.97.62:46323)
0923 20:54:47.747028 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:47.747033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:47.747533 (+ 500us) server_negotiation.cc:407] Connection header received
0923 20:54:47.747736 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:47.747741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:47.747805 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:47.747910 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:47.748267 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45530: BlockingRecv error: recv got EOF from 127.0.0.1:45530 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:54:47.749526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10307
I20260923 20:54:47.753609 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:47.750483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.0.97.1:34369)
0923 20:54:47.751002 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:47.751013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:47.751319 (+ 306us) server_negotiation.cc:407] Connection header received
0923 20:54:47.751516 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:47.751525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:47.751759 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:47.751915 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:47.752894 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:47.752900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:47.752912 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:47.752930 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:47.752969 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:47.752983 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:47.753127 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:47.753230 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 20:54:47.753342 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
I20260923 20:54:47.754990 10429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:47.834964 10434 heartbeater.cc:503] Master 127.0.97.62:46323 was elected leader, sending a full tablet report...
I20260923 20:54:47.836561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10307
I20260923 20:54:47.853539 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10438
I20260923 20:54:47.857445 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:47.854611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.97.2:45395)
0923 20:54:47.855110 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:47.855117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:47.855142 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:54:47.855228 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:47.855238 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:47.855520 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:47.855694 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:47.856748 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:47.856754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:47.856766 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:47.856783 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:47.856825 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:47.856838 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:47.856990 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:47.857072 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 20:54:47.857176 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":246,"threads_started":1}
I20260923 20:54:47.858822 10560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:47.943687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10438
I20260923 20:54:47.960052 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10568
I20260923 20:54:47.964148 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:47.961160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.97.3:32913)
0923 20:54:47.961868 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:47.961876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:47.961907 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:54:47.961968 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:47.961974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:47.962192 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:47.962327 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:47.963420 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:47.963427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:47.963439 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:54:47.963455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:47.963492 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:54:47.963506 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:47.963649 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:47.963698 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 20:54:47.963801 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":337,"threads_started":1}
I20260923 20:54:47.965377 10690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:48.047030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10568
I20260923 20:54:48.062932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10217
I20260923 20:54:48.064132 10278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:48.166414 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10217
2026-09-23T20:54:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2472 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:54:48 dist-test-slave-plcg krb5kdc[10713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:54:48 dist-test-slave-plcg krb5kdc[10713](info): set up 2 sockets
Sep 23 20:54:48 dist-test-slave-plcg krb5kdc[10713](info): commencing operation
krb5kdc: starting...
W20260923 20:54:49.282770 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:54:49 dist-test-slave-plcg krb5kdc[10713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:54:49.530460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38369
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:49.871158 10726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:49.871474 10726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:49.871656 10726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:49.879098 10726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:54:49.879221 10726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:49.879338 10726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:49.879443 10726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:54:49.879530 10726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:54:49.895671 10726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:49.897595 10726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:49.899101 10726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:49.911293 10731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:49.911226 10734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:49.911332 10732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:49.912505 10726 server_base.cc:1034] running on GCE node
I20260923 20:54:49.913358 10726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:49.915036 10726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:49.916265 10726 hybrid_clock.cc:648] HybridClock initialized: now 1790196889916214 us; error 42 us; skew 500 ppm
I20260923 20:54:49.916604 10726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:49 dist-test-slave-plcg krb5kdc[10713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:49.919927 10726 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:54:49.921612 10726 webserver.cc:460] Webserver started at http://127.0.97.62:33249/ using document root <none> and password file <none>
I20260923 20:54:49.922428 10726 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:49.922544 10726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:49.922808 10726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:49.926123 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "a8858464f2464b6889661d3e6cee22b9"
format_stamp: "Formatted at 2026-09-23 20:54:49 on dist-test-slave-plcg"
server_key: "2616bfaf21a9fa3c01d07d17e9a5e800"
server_key_iv: "b9e2e4ee6877103af7c91d27b43ab1fb"
server_key_version: "encryptionkey@0"
I20260923 20:54:49.926935 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "a8858464f2464b6889661d3e6cee22b9"
format_stamp: "Formatted at 2026-09-23 20:54:49 on dist-test-slave-plcg"
server_key: "2616bfaf21a9fa3c01d07d17e9a5e800"
server_key_iv: "b9e2e4ee6877103af7c91d27b43ab1fb"
server_key_version: "encryptionkey@0"
I20260923 20:54:49.932628 10726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:54:49.936481 10741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:49.938169 10726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:54:49.938342 10726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "a8858464f2464b6889661d3e6cee22b9"
format_stamp: "Formatted at 2026-09-23 20:54:49 on dist-test-slave-plcg"
server_key: "2616bfaf21a9fa3c01d07d17e9a5e800"
server_key_iv: "b9e2e4ee6877103af7c91d27b43ab1fb"
server_key_version: "encryptionkey@0"
I20260923 20:54:49.938520 10726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:49.971984 10726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:49.976756 10726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:49.977010 10726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:49.996402 10726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38369
I20260923 20:54:49.996511 10792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38369 every 8 connection(s)
I20260923 20:54:49.997882 10726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:54:50.001798 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:50.004914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10726
I20260923 20:54:50.005149 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:54:50.005990 388 external_mini_cluster.cc:1428] Setting key 0c3c95850b83d0162bfa573dc38fc22a
I20260923 20:54:50.015734 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9: Bootstrap starting.
I20260923 20:54:50.019132 10793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:50.020238 10793 log.cc:824] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:54:50 dist-test-slave-plcg krb5kdc[10713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:50.023554 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9: No bootstrap required, opened a new log
I20260923 20:54:50.030658 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:50.009090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.0.97.62:38369)
0923 20:54:50.009985 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:50.010000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:50.010047 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:54:50.011814 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:50.011850 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:50.012305 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:50.012777 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:50.014572 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:50.015975 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:50.017338 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:50.018141 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:50.018343 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:50.022707 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:50.022716 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:50.022733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:50.022769 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:50.026104 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:50.026898 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:50.026909 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:50.026916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:50.027105 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:50.027514 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:50.027521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:50.027524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:50.028140 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:50.028474 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:50.028950 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 20:54:50.029417 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":344,"threads_started":1}
I20260923 20:54:50.032073 10793 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858464f2464b6889661d3e6cee22b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38369 } }
I20260923 20:54:50.032475 10793 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:50.032604 10793 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8858464f2464b6889661d3e6cee22b9, State: Initialized, Role: FOLLOWER
I20260923 20:54:50.033210 10793 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858464f2464b6889661d3e6cee22b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38369 } }
I20260923 20:54:50.033456 10793 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:54:50.033603 10793 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:54:50.033792 10793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:50.036147 10793 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858464f2464b6889661d3e6cee22b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38369 } }
I20260923 20:54:50.036835 10793 leader_election.cc:302] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8858464f2464b6889661d3e6cee22b9; no voters:
I20260923 20:54:50.037744 10793 leader_election.cc:288] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:54:50.038079 10798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:50.039036 10798 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [term 1 LEADER]: Becoming Leader. State: Replica: a8858464f2464b6889661d3e6cee22b9, State: Running, Role: LEADER
I20260923 20:54:50.039628 10798 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858464f2464b6889661d3e6cee22b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38369 } }
I20260923 20:54:50.040292 10793 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:54:50.042794 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8858464f2464b6889661d3e6cee22b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858464f2464b6889661d3e6cee22b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38369 } } }
I20260923 20:54:50.043215 10800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8858464f2464b6889661d3e6cee22b9. Latest consensus state: current_term: 1 leader_uuid: "a8858464f2464b6889661d3e6cee22b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8858464f2464b6889661d3e6cee22b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38369 } } }
I20260923 20:54:50.043900 10800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:50.043898 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:54:50.047581 10807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:54:50.053587 10807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:54:50.064775 10807 catalog_manager.cc:1347] Generated new cluster ID: c6224cd6523b4425be736b35356464ce
I20260923 20:54:50.064930 10807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:54:50.118480 10807 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:54:50.119966 10807 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:54:50.126013 10807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9: Generated new TSK 0
I20260923 20:54:50.126942 10807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:54:50.189950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38369
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:50.541447 10819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:50.541934 10819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:50.542116 10819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:50.550215 10819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:50.550345 10819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:50.550559 10819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:54:50.560870 10819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:50.562317 10819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:50.563686 10819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:50.578593 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:50.578547 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:50.578553 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:50.578945 10819 server_base.cc:1034] running on GCE node
I20260923 20:54:50.579955 10819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:50.581046 10819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:50.582261 10819 hybrid_clock.cc:648] HybridClock initialized: now 1790196890582209 us; error 44 us; skew 500 ppm
I20260923 20:54:50.582639 10819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:50 dist-test-slave-plcg krb5kdc[10713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:50.591918 10819 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:54:50.593737 10819 webserver.cc:460] Webserver started at http://127.0.97.1:32929/ using document root <none> and password file <none>
I20260923 20:54:50.594585 10819 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:50.594700 10819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:50.594980 10819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:50.598272 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b"
format_stamp: "Formatted at 2026-09-23 20:54:50 on dist-test-slave-plcg"
server_key: "f7107c7bb6e651b8b7ebf7cd9f593a11"
server_key_iv: "8b49a1893c402748578e315c6eb8ef61"
server_key_version: "encryptionkey@0"
I20260923 20:54:50.599104 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b"
format_stamp: "Formatted at 2026-09-23 20:54:50 on dist-test-slave-plcg"
server_key: "f7107c7bb6e651b8b7ebf7cd9f593a11"
server_key_iv: "8b49a1893c402748578e315c6eb8ef61"
server_key_version: "encryptionkey@0"
I20260923 20:54:50.604784 10819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:54:50.608750 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:50.610451 10819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:54:50.610664 10819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b"
format_stamp: "Formatted at 2026-09-23 20:54:50 on dist-test-slave-plcg"
server_key: "f7107c7bb6e651b8b7ebf7cd9f593a11"
server_key_iv: "8b49a1893c402748578e315c6eb8ef61"
server_key_version: "encryptionkey@0"
I20260923 20:54:50.610898 10819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:50.650409 10819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:50.655052 10819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:50.655376 10819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:50.656436 10819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:50.657904 10819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:50.658013 10819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:50.658164 10819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:50.658236 10819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:50.693205 10819 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33667
I20260923 20:54:50.693306 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33667 every 8 connection(s)
I20260923 20:54:50.694635 10819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:54:50.698036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10819
I20260923 20:54:50.698304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:54:50.699281 388 external_mini_cluster.cc:1428] Setting key dd3a56519ccc7b929dc1dde7b573103b
Sep 23 20:54:50 dist-test-slave-plcg krb5kdc[10713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:50.718914 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:50.697581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40561 (local address 127.0.97.62:38369)
0923 20:54:50.698251 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:50.698261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:50.699246 (+ 985us) server_negotiation.cc:407] Connection header received
0923 20:54:50.701388 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:50.701397 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:50.701486 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:50.701632 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:50.704064 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:50.704813 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:50.707467 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:50.707991 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:50.708129 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:50.712261 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:50.712270 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:50.712278 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:50.712328 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:50.715626 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:50.716465 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:50.716470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:50.716473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:50.716589 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:50.717027 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:50.717032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:50.717036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:50.717306 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:50.717468 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:50.718292 (+ 824us) server_negotiation.cc:299] Negotiation successful
0923 20:54:50.718533 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":384,"threads_started":1}
I20260923 20:54:50.719959 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:50.698081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38369 (local address 127.0.97.1:40561)
0923 20:54:50.699051 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:50.699097 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:50.701017 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:50.702044 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:50.702058 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:50.702896 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:50.703818 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:50.703854 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:50.704995 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:50.705003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:50.707272 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:50.707294 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:50.708263 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:50.708273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:50.708640 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:50.709496 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:50.709533 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:50.711965 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:50.715800 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:50.715812 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:50.715827 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:50.716310 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:50.716701 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:50.716706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:50.716710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:50.716905 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:50.717443 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:50.717452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:50.717898 (+ 446us) client_negotiation.cc:769] Sending connection context
0923 20:54:50.718187 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 20:54:50.718759 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":435,"threads_started":1}
I20260923 20:54:50.721742 10947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38369
I20260923 20:54:50.722124 10947 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:50.722910 10947 heartbeater.cc:511] Master 127.0.97.62:38369 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:54:50.725310 10758 ts_manager.cc:194] Registered new tserver with Master: dadf4be8a6cd4151b3de1f2403a1bf9b (127.0.97.1:33667)
I20260923 20:54:50.727072 10758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:54:50.757642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38369
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:51.105425 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:51.105742 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:51.105933 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:51.113643 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:51.113775 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:51.113999 10953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:54:51.124475 10953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:51.125972 10953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:51.127532 10953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:51.142590 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:51.143265 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:51.143266 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:51.144505 10953 server_base.cc:1034] running on GCE node
I20260923 20:54:51.145223 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:51.146282 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:51.147536 10953 hybrid_clock.cc:648] HybridClock initialized: now 1790196891147485 us; error 39 us; skew 500 ppm
I20260923 20:54:51.147925 10953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:51 dist-test-slave-plcg krb5kdc[10713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:51.151921 10953 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:54:51.153702 10953 webserver.cc:460] Webserver started at http://127.0.97.2:42067/ using document root <none> and password file <none>
I20260923 20:54:51.154528 10953 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:51.154690 10953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:51.155017 10953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:51.158278 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "63bc49ab88094fdabc68ba8bf0500ee5"
format_stamp: "Formatted at 2026-09-23 20:54:51 on dist-test-slave-plcg"
server_key: "1bc7b4abe18607a9ff233633881ca273"
server_key_iv: "4cfe3f2ba8350d75a74716a2a14794be"
server_key_version: "encryptionkey@0"
I20260923 20:54:51.159144 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "63bc49ab88094fdabc68ba8bf0500ee5"
format_stamp: "Formatted at 2026-09-23 20:54:51 on dist-test-slave-plcg"
server_key: "1bc7b4abe18607a9ff233633881ca273"
server_key_iv: "4cfe3f2ba8350d75a74716a2a14794be"
server_key_version: "encryptionkey@0"
I20260923 20:54:51.165262 10953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 20:54:51.170004 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:51.171847 10953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:54:51.172089 10953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "63bc49ab88094fdabc68ba8bf0500ee5"
format_stamp: "Formatted at 2026-09-23 20:54:51 on dist-test-slave-plcg"
server_key: "1bc7b4abe18607a9ff233633881ca273"
server_key_iv: "4cfe3f2ba8350d75a74716a2a14794be"
server_key_version: "encryptionkey@0"
I20260923 20:54:51.172346 10953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:51.200979 10953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:51.206395 10953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:51.206724 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:51.207789 10953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:51.209573 10953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:51.209683 10953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:51.209867 10953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:51.209945 10953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 20:54:51.246603 10953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38483
I20260923 20:54:51.246661 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38483 every 8 connection(s)
I20260923 20:54:51.247928 10953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:54:51.252293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 10953
I20260923 20:54:51.252507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:54:51.252950 388 external_mini_cluster.cc:1428] Setting key 31ed9e81cbac2d83d5091c19a2368859
Sep 23 20:54:51 dist-test-slave-plcg krb5kdc[10713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:51.269374 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.250705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47465 (local address 127.0.97.62:38369)
0923 20:54:51.251170 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:51.251175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:51.252071 (+ 896us) server_negotiation.cc:407] Connection header received
0923 20:54:51.254207 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:51.254212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:51.254296 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:51.254425 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:51.256187 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.256937 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.258712 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.259383 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.259579 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.263323 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:51.263329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:51.263332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:51.263368 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:51.266146 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:51.266993 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:51.266997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:51.267000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:51.267128 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:51.267584 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:51.267591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:51.267599 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:51.267859 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:51.268023 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:51.268753 (+ 730us) server_negotiation.cc:299] Negotiation successful
0923 20:54:51.268990 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260923 20:54:51.270519 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.251149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38369 (local address 127.0.97.2:47465)
0923 20:54:51.251905 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:51.251948 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:51.253899 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:51.254740 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:51.254754 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:51.255369 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:51.255974 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.256009 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.257108 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.257120 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.258535 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.258550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.259686 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.259694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.259961 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.260706 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:51.260731 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:51.263043 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:51.266336 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:51.266346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:51.266375 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:51.266838 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:51.267246 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:51.267251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:51.267255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:51.267457 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:51.267993 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:51.268002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:51.268380 (+ 378us) client_negotiation.cc:769] Sending connection context
0923 20:54:51.268661 (+ 281us) client_negotiation.cc:240] Negotiation successful
0923 20:54:51.269245 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":315,"threads_started":1}
I20260923 20:54:51.272701 11081 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38369
I20260923 20:54:51.273083 11081 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:51.273957 11081 heartbeater.cc:511] Master 127.0.97.62:38369 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:54:51.275843 10758 ts_manager.cc:194] Registered new tserver with Master: 63bc49ab88094fdabc68ba8bf0500ee5 (127.0.97.2:38483)
I20260923 20:54:51.276805 10758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:54:51.311102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38369
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:54:51.655081 11087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:54:51.655400 11087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:54:51.655496 11087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:54:51.662922 11087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:54:51.663106 11087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:54:51.663290 11087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:54:51.673265 11087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:54:51.674844 11087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:54:51.676287 11087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:54:51.690491 11093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:51.690506 11092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:54:51.690441 11095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:54:51.691310 11087 server_base.cc:1034] running on GCE node
I20260923 20:54:51.692030 11087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:54:51.694214 11087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:54:51.695511 11087 hybrid_clock.cc:648] HybridClock initialized: now 1790196891695446 us; error 57 us; skew 500 ppm
I20260923 20:54:51.695888 11087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:54:51 dist-test-slave-plcg krb5kdc[10713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:54:51.699312 11087 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:54:51.700879 11087 webserver.cc:460] Webserver started at http://127.0.97.3:42453/ using document root <none> and password file <none>
I20260923 20:54:51.701642 11087 fs_manager.cc:359] Metadata directory not provided
I20260923 20:54:51.701797 11087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:54:51.702147 11087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:54:51.705358 11087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "1386e6ee968845ba83b95f26b2420725"
format_stamp: "Formatted at 2026-09-23 20:54:51 on dist-test-slave-plcg"
server_key: "e0403e428a01a79d502bba843dac81b2"
server_key_iv: "82ee921f4df6358b08364b62f064e57d"
server_key_version: "encryptionkey@0"
I20260923 20:54:51.706247 11087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1386e6ee968845ba83b95f26b2420725"
format_stamp: "Formatted at 2026-09-23 20:54:51 on dist-test-slave-plcg"
server_key: "e0403e428a01a79d502bba843dac81b2"
server_key_iv: "82ee921f4df6358b08364b62f064e57d"
server_key_version: "encryptionkey@0"
I20260923 20:54:51.711905 11087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:54:51.715943 11102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:51.717446 11087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:54:51.717659 11087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "1386e6ee968845ba83b95f26b2420725"
format_stamp: "Formatted at 2026-09-23 20:54:51 on dist-test-slave-plcg"
server_key: "e0403e428a01a79d502bba843dac81b2"
server_key_iv: "82ee921f4df6358b08364b62f064e57d"
server_key_version: "encryptionkey@0"
I20260923 20:54:51.717932 11087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:54:51.730883 10947 heartbeater.cc:503] Master 127.0.97.62:38369 was elected leader, sending a full tablet report...
I20260923 20:54:51.739576 11087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:54:51.744438 11087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:54:51.744752 11087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:54:51.745745 11087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:54:51.747128 11087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:54:51.747239 11087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:51.747423 11087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:54:51.747498 11087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:54:51.781343 11087 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40073
I20260923 20:54:51.781383 11214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40073 every 8 connection(s)
I20260923 20:54:51.782723 11087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:54:51.785537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 11087
I20260923 20:54:51.785737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:54:51.786187 388 external_mini_cluster.cc:1428] Setting key ca6a1468a02b8db77a0190ae1786ab98
Sep 23 20:54:51 dist-test-slave-plcg krb5kdc[10713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:54:51.803571 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.785248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47559 (local address 127.0.97.62:38369)
0923 20:54:51.785784 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:51.785790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:51.787048 (+ 1258us) server_negotiation.cc:407] Connection header received
0923 20:54:51.789465 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:51.789469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:51.789540 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:51.789640 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:51.791273 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.792033 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.793560 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.794004 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.794102 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.797710 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:51.797715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:51.797718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:51.797760 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:51.800349 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:51.801150 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:51.801158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:51.801164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:51.801289 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:51.801741 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:51.801747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:51.801751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:51.802084 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:51.802239 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:51.802962 (+ 723us) server_negotiation.cc:299] Negotiation successful
0923 20:54:51.803185 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":241,"threads_started":1}
I20260923 20:54:51.804555 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.785988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38369 (local address 127.0.97.3:47559)
0923 20:54:51.786858 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:51.786896 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:51.789104 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:51.789931 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:51.789940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:51.790522 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:54:51.791106 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.791124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.792165 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.792169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.793428 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.793441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.794228 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.794233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.794446 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.795179 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:54:51.795207 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:54:51.797469 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:54:51.800553 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:51.800561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:51.800576 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:51.801020 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:51.801411 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:54:51.801416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:54:51.801420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:54:51.801625 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:54:51.802219 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:54:51.802228 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:54:51.802590 (+ 362us) client_negotiation.cc:769] Sending connection context
0923 20:54:51.802869 (+ 279us) client_negotiation.cc:240] Negotiation successful
0923 20:54:51.803439 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":357,"threads_started":1}
I20260923 20:54:51.806002 11215 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38369
I20260923 20:54:51.806324 11215 heartbeater.cc:464] Registering TS with master...
I20260923 20:54:51.807085 11215 heartbeater.cc:511] Master 127.0.97.62:38369 requested a full tablet report, sending...
I20260923 20:54:51.808689 10758 ts_manager.cc:194] Registered new tserver with Master: 1386e6ee968845ba83b95f26b2420725 (127.0.97.3:40073)
I20260923 20:54:51.809514 10758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47559
I20260923 20:54:51.813895 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:54:51.832131 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.818837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.97.62:38369)
0923 20:54:51.819020 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:51.819025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:51.819581 (+ 556us) server_negotiation.cc:407] Connection header received
0923 20:54:51.819738 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:51.819742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:51.819846 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:51.819935 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:51.821764 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.822728 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.824070 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.824478 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.824575 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.826929 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:51.826934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:51.826938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:51.826975 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:51.829404 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:51.830179 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:51.830183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:51.830185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:51.830318 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:51.830760 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:51.830764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:51.830766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:51.831117 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:51.831302 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:51.831617 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 20:54:51.831782 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:54:51.836483 10758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:54:51.843883 10758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:54:51.869516 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.856323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52022)
0923 20:54:51.857534 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:51.857584 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:51.857844 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:51.858691 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:51.858699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:51.858731 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:51.859246 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.859274 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.861968 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.861977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.865306 (+ 3329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.865330 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.868225 (+ 2895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.868235 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.868541 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.868628 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 20:54:51.868774 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 20:54:51.869090 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":1909,"threads_started":1}
I20260923 20:54:51.871613 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.856995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.0.97.1:33667)
0923 20:54:51.857963 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:51.857973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:51.858006 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:54:51.858107 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:51.858114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:51.858406 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:51.858657 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:51.859547 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.861696 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.865641 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.867870 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.870665 (+ 2795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.870970 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:51.871056 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 20:54:51.871242 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":1984,"threads_started":1}
I20260923 20:54:51.876102 10882 tablet_service.cc:1467] Processing CreateTablet for tablet bb5b8a25bbc0451497fbc36adf01f5b5 (DEFAULT_TABLE table=test-table [id=c087128e795c45fba3816476faee0691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:51.877656 10882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5b8a25bbc0451497fbc36adf01f5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:51.880348 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.856524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40073 (local address 127.0.0.1:59088)
0923 20:54:51.858534 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:51.858554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:51.858691 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:51.870595 (+ 11904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:51.870599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:51.870622 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:51.871002 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.871014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.873203 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.873209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.877118 (+ 3909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.877134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.879477 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.879484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.879684 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.879722 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:54:51.879830 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:54:51.879984 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1895,"thread_start_us":1392,"threads_started":1}
I20260923 20:54:51.881187 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.857611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.0.97.3:40073)
0923 20:54:51.869635 (+ 12024us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:51.869648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:51.869693 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:54:51.869805 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:51.869835 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:51.870149 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:51.871209 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:51.871511 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.873034 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.877314 (+ 4280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.878919 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.879067 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.879311 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:51.880640 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0923 20:54:51.880827 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11812,"thread_start_us":12676,"threads_started":1}
I20260923 20:54:51.881548 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.858808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38483 (local address 127.0.0.1:58230)
0923 20:54:51.870370 (+ 11562us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:51.870389 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:51.870480 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:51.873663 (+ 3183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:51.873668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:51.873697 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:51.875714 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.875726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.877568 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.877574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.878852 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:51.878866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.880841 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.880846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:51.881035 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.881071 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:54:51.881126 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 20:54:51.881228 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11453,"thread_start_us":16645,"threads_started":1}
I20260923 20:54:51.882545 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:51.859213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.97.2:38483)
0923 20:54:51.873008 (+ 13795us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:51.873014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:51.873039 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:54:51.873149 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:51.873157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:51.873433 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:51.873652 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:51.875886 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.877430 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.879032 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:51.880210 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:51.880337 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:51.881999 (+ 1662us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:51.882071 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 20:54:51.882201 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13624,"thread_start_us":13474,"threads_started":1}
I20260923 20:54:51.885682 11150 tablet_service.cc:1467] Processing CreateTablet for tablet bb5b8a25bbc0451497fbc36adf01f5b5 (DEFAULT_TABLE table=test-table [id=c087128e795c45fba3816476faee0691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:51.891427 11016 tablet_service.cc:1467] Processing CreateTablet for tablet bb5b8a25bbc0451497fbc36adf01f5b5 (DEFAULT_TABLE table=test-table [id=c087128e795c45fba3816476faee0691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:54:51.893139 11016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5b8a25bbc0451497fbc36adf01f5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:51.893921 11234 tablet_bootstrap.cc:492] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Bootstrap starting.
I20260923 20:54:51.894770 11150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5b8a25bbc0451497fbc36adf01f5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:54:51.897684 11234 tablet_bootstrap.cc:654] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:51.899113 11234 log.cc:824] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:51.902627 11234 tablet_bootstrap.cc:492] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: No bootstrap required, opened a new log
I20260923 20:54:51.902935 11234 ts_tablet_manager.cc:1397] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:54:51.905802 11236 tablet_bootstrap.cc:492] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Bootstrap starting.
I20260923 20:54:51.910341 11237 tablet_bootstrap.cc:492] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Bootstrap starting.
I20260923 20:54:51.910427 11236 tablet_bootstrap.cc:654] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:51.911919 11236 log.cc:824] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:51.913726 11234 raft_consensus.cc:348] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:51.914126 11237 tablet_bootstrap.cc:654] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Neither blocks nor log segments found. Creating new log.
I20260923 20:54:51.914193 11234 raft_consensus.cc:374] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:51.914336 11234 raft_consensus.cc:729] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dadf4be8a6cd4151b3de1f2403a1bf9b, State: Initialized, Role: FOLLOWER
I20260923 20:54:51.915050 11234 consensus_queue.cc:260] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:51.915596 11237 log.cc:824] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Log is configured to *not* fsync() on all Append() calls
I20260923 20:54:51.915671 11236 tablet_bootstrap.cc:492] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: No bootstrap required, opened a new log
I20260923 20:54:51.915949 11236 ts_tablet_manager.cc:1397] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260923 20:54:51.917126 11234 ts_tablet_manager.cc:1428] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 20:54:51.920821 11237 tablet_bootstrap.cc:492] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: No bootstrap required, opened a new log
I20260923 20:54:51.921209 11237 ts_tablet_manager.cc:1397] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260923 20:54:51.927023 11236 raft_consensus.cc:348] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:51.927544 11236 raft_consensus.cc:374] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:51.927640 11236 raft_consensus.cc:729] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bc49ab88094fdabc68ba8bf0500ee5, State: Initialized, Role: FOLLOWER
I20260923 20:54:51.928334 11236 consensus_queue.cc:260] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:51.929553 11081 heartbeater.cc:503] Master 127.0.97.62:38369 was elected leader, sending a full tablet report...
I20260923 20:54:51.930341 11236 ts_tablet_manager.cc:1428] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260923 20:54:51.931752 11237 raft_consensus.cc:348] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:51.932152 11237 raft_consensus.cc:374] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:54:51.932268 11237 raft_consensus.cc:729] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1386e6ee968845ba83b95f26b2420725, State: Initialized, Role: FOLLOWER
I20260923 20:54:51.932881 11237 consensus_queue.cc:260] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:51.934084 11215 heartbeater.cc:503] Master 127.0.97.62:38369 was elected leader, sending a full tablet report...
I20260923 20:54:51.934788 11237 ts_tablet_manager.cc:1428] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260923 20:54:51.947250 10948 tablet.cc:2369] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:54:52.000203 11082 tablet.cc:2369] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:54:52.034437 11216 tablet.cc:2369] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:54:52.084630 11242 raft_consensus.cc:482] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:54:52.084973 11242 raft_consensus.cc:504] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:52.086823 11242 leader_election.cc:288] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dadf4be8a6cd4151b3de1f2403a1bf9b (127.0.97.1:33667), 63bc49ab88094fdabc68ba8bf0500ee5 (127.0.97.2:38483)
I20260923 20:54:52.092840 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.087289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49359 (local address 127.0.97.1:33667)
0923 20:54:52.087485 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:52.087491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:52.087685 (+ 194us) server_negotiation.cc:407] Connection header received
0923 20:54:52.087789 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:52.087796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:52.087889 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:52.088003 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:52.088684 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.089405 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.091161 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.091930 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.092076 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.092175 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:52.092447 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 20:54:52.092571 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:54:52.092882 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.087116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.97.3:49359)
0923 20:54:52.087486 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.087507 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:52.087657 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:52.088044 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:52.088048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:52.088076 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:52.088297 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:52.088310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.089541 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.089548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:52.090985 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:52.091003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.092102 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.092108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:52.092285 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.092312 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 20:54:52.092421 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:54:52.092572 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260923 20:54:52.093251 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.087372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49171 (local address 127.0.97.2:38483)
0923 20:54:52.087607 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:52.087616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:52.088293 (+ 677us) server_negotiation.cc:407] Connection header received
0923 20:54:52.088426 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:52.088430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:52.088485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:52.088558 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:54:52.089036 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.089741 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.091552 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.092331 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.092503 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.092595 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:52.092858 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 20:54:52.092987 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:54:52.093312 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.087265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38483 (local address 127.0.97.3:49171)
0923 20:54:52.088176 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.088193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:54:52.088314 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:54:52.088585 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:54:52.088588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:54:52.088606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:54:52.088874 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:52.088881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.089915 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.089925 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:52.091403 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:54:52.091415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.092548 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.092554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:54:52.092705 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.092727 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:54:52.092833 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:54:52.092970 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":454,"threads_started":1}
I20260923 20:54:52.094352 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b8a25bbc0451497fbc36adf01f5b5" candidate_uuid: "1386e6ee968845ba83b95f26b2420725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" is_pre_election: true
I20260923 20:54:52.094470 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b8a25bbc0451497fbc36adf01f5b5" candidate_uuid: "1386e6ee968845ba83b95f26b2420725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" is_pre_election: true
I20260923 20:54:52.094933 10902 raft_consensus.cc:2454] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1386e6ee968845ba83b95f26b2420725 in term 0.
I20260923 20:54:52.094951 11036 raft_consensus.cc:2454] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1386e6ee968845ba83b95f26b2420725 in term 0.
I20260923 20:54:52.095602 11106 leader_election.cc:302] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1386e6ee968845ba83b95f26b2420725, 63bc49ab88094fdabc68ba8bf0500ee5, dadf4be8a6cd4151b3de1f2403a1bf9b; no voters:
I20260923 20:54:52.096038 11242 raft_consensus.cc:2781] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:54:52.096181 11242 raft_consensus.cc:482] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:54:52.096280 11242 raft_consensus.cc:3037] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:52.098613 11242 raft_consensus.cc:504] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:52.099272 11242 leader_election.cc:288] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [CANDIDATE]: Term 1 election: Requested vote from peers dadf4be8a6cd4151b3de1f2403a1bf9b (127.0.97.1:33667), 63bc49ab88094fdabc68ba8bf0500ee5 (127.0.97.2:38483)
I20260923 20:54:52.099783 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b8a25bbc0451497fbc36adf01f5b5" candidate_uuid: "1386e6ee968845ba83b95f26b2420725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63bc49ab88094fdabc68ba8bf0500ee5"
I20260923 20:54:52.099804 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5b8a25bbc0451497fbc36adf01f5b5" candidate_uuid: "1386e6ee968845ba83b95f26b2420725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b"
I20260923 20:54:52.100006 11036 raft_consensus.cc:3037] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:52.100006 10902 raft_consensus.cc:3037] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:54:52.103235 11036 raft_consensus.cc:2454] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1386e6ee968845ba83b95f26b2420725 in term 1.
I20260923 20:54:52.103235 10902 raft_consensus.cc:2454] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1386e6ee968845ba83b95f26b2420725 in term 1.
I20260923 20:54:52.103801 11104 leader_election.cc:302] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1386e6ee968845ba83b95f26b2420725, 63bc49ab88094fdabc68ba8bf0500ee5; no voters:
I20260923 20:54:52.104203 11242 raft_consensus.cc:2781] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:54:52.109198 11242 raft_consensus.cc:686] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 1 LEADER]: Becoming Leader. State: Replica: 1386e6ee968845ba83b95f26b2420725, State: Running, Role: LEADER
I20260923 20:54:52.109942 11242 consensus_queue.cc:237] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } }
I20260923 20:54:52.115538 10757 catalog_manager.cc:5495] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1386e6ee968845ba83b95f26b2420725 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1386e6ee968845ba83b95f26b2420725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1386e6ee968845ba83b95f26b2420725" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 } health_report { overall_health: UNKNOWN } } }
I20260923 20:54:52.166524 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.159658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.97.3:40073)
0923 20:54:52.159912 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:52.159920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:52.159948 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:54:52.160079 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:52.160086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:52.160227 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:52.160439 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:54:52.160824 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.162150 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.163499 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.164318 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.164469 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.164964 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:54:52.165764 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:54:52.165970 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:52.166032 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 20:54:52.166141 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:54:52.175161 10902 raft_consensus.cc:1261] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 1 FOLLOWER]: Refusing update from remote peer 1386e6ee968845ba83b95f26b2420725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:52.175345 11036 raft_consensus.cc:1261] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 1 FOLLOWER]: Refusing update from remote peer 1386e6ee968845ba83b95f26b2420725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:54:52.176270 11242 consensus_queue.cc:1035] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dadf4be8a6cd4151b3de1f2403a1bf9b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:52.177001 11244 consensus_queue.cc:1035] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63bc49ab88094fdabc68ba8bf0500ee5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:54:52.189807 11248 mvcc.cc:205] Tried to move back new op lower bound from 7332646470337957888 to 7332646470086676480. Current Snapshot: MvccSnapshot[applied={T|T < 7332646470337957888}]
I20260923 20:54:52.209131 10757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:52.209576 10757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:54:52.215175 10757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8858464f2464b6889661d3e6cee22b9: Sending DeleteTablet for 3 replicas of tablet bb5b8a25bbc0451497fbc36adf01f5b5
I20260923 20:54:52.216449 10882 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5b8a25bbc0451497fbc36adf01f5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52022
I20260923 20:54:52.222868 11150 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5b8a25bbc0451497fbc36adf01f5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59088
I20260923 20:54:52.224833 11149 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5b8a25bbc0451497fbc36adf01f5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59088
I20260923 20:54:52.225495 11016 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5b8a25bbc0451497fbc36adf01f5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:54:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58230
I20260923 20:54:52.228523 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10819
I20260923 20:54:52.229228 11256 tablet_replica.cc:330] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: stopping tablet replica
I20260923 20:54:52.230340 11256 raft_consensus.cc:2229] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:54:52.234072 11258 tablet_replica.cc:330] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: stopping tablet replica
I20260923 20:54:52.234547 11258 raft_consensus.cc:2229] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:52.234994 11256 raft_consensus.cc:2258] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:52.235057 11258 raft_consensus.cc:2258] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:52.238263 11256 ts_tablet_manager.cc:1905] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:52.238862 11258 ts_tablet_manager.cc:1905] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:52.241137 11255 tablet_replica.cc:330] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: stopping tablet replica
I20260923 20:54:52.241731 11255 raft_consensus.cc:2229] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:54:52.243599 11255 raft_consensus.cc:2258] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:54:52.250345 11258 ts_tablet_manager.cc:1918] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:52.250597 11258 log.cc:1196] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/bb5b8a25bbc0451497fbc36adf01f5b5
I20260923 20:54:52.251207 11258 ts_tablet_manager.cc:1939] T bb5b8a25bbc0451497fbc36adf01f5b5 P 63bc49ab88094fdabc68ba8bf0500ee5: Deleting consensus metadata
I20260923 20:54:52.253226 10743 catalog_manager.cc:4841] TS 63bc49ab88094fdabc68ba8bf0500ee5 (127.0.97.2:38483): tablet bb5b8a25bbc0451497fbc36adf01f5b5 (table test-table [id=c087128e795c45fba3816476faee0691]) successfully deleted
I20260923 20:54:52.254989 11256 ts_tablet_manager.cc:1918] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:52.255252 11256 log.cc:1196] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/bb5b8a25bbc0451497fbc36adf01f5b5
I20260923 20:54:52.255916 11256 ts_tablet_manager.cc:1939] T bb5b8a25bbc0451497fbc36adf01f5b5 P 1386e6ee968845ba83b95f26b2420725: Deleting consensus metadata
Sep 23 20:54:52 dist-test-slave-plcg krb5kdc[10713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:54:52.257946 10743 catalog_manager.cc:4841] TS 1386e6ee968845ba83b95f26b2420725 (127.0.97.3:40073): tablet bb5b8a25bbc0451497fbc36adf01f5b5 (table test-table [id=c087128e795c45fba3816476faee0691]) successfully deleted
W20260923 20:54:52.261679 10743 catalog_manager.cc:4808] TS 1386e6ee968845ba83b95f26b2420725 (127.0.97.3:40073): delete failed for tablet bb5b8a25bbc0451497fbc36adf01f5b5 because the tablet was not found. No further retry: Not found: Tablet not found: bb5b8a25bbc0451497fbc36adf01f5b5
I20260923 20:54:52.264765 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.229913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.0.97.1:33667)
0923 20:54:52.230144 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:52.230153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:52.233307 (+ 3154us) server_negotiation.cc:407] Connection header received
0923 20:54:52.233393 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:52.233401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:52.233540 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:52.233658 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:52.242667 (+ 9009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.243912 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.248135 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.248797 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.248910 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.257514 (+ 8604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:52.257532 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:52.257549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:52.257598 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:52.260943 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:52.262104 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:52.262117 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:52.262123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:52.262296 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:52.262795 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:52.262800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:52.262804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:52.263314 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:52.263608 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:52.264147 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 20:54:52.264394 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:54:52.266422 10942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:52.274698 11255 ts_tablet_manager.cc:1905] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:54:52.388731 11255 ts_tablet_manager.cc:1918] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:54:52.388942 11255 log.cc:1196] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/bb5b8a25bbc0451497fbc36adf01f5b5
I20260923 20:54:52.389544 11255 ts_tablet_manager.cc:1939] T bb5b8a25bbc0451497fbc36adf01f5b5 P dadf4be8a6cd4151b3de1f2403a1bf9b: Deleting consensus metadata
I20260923 20:54:52.389858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10819
W20260923 20:54:52.411331 10745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33667 (error 108)
I20260923 20:54:52.412374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10953
Sep 23 20:54:52 dist-test-slave-plcg krb5kdc[10713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:54:52.435535 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.424537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52036)
0923 20:54:52.435128 (+ 10591us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.435285 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8817,"thread_start_us":7647,"threads_started":1}
W20260923 20:54:52.435995 10745 catalog_manager.cc:4584] TS dadf4be8a6cd4151b3de1f2403a1bf9b (127.0.97.1:33667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb5b8a25bbc0451497fbc36adf01f5b5: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
I20260923 20:54:52.438314 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.414392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.97.2:38483)
0923 20:54:52.414643 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:52.414652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:52.414676 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:54:52.414758 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:52.414766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:52.414884 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:52.415010 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:52.417385 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.418995 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.421751 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.422794 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.423001 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.430726 (+ 7725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:52.430754 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:52.430782 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:52.430857 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:52.434841 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:52.435842 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:52.435853 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:52.435856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:52.436055 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:52.436504 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:52.436509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:52.436513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:52.436910 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:52.437211 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:52.437538 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 20:54:52.437757 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:54:52.439049 11076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:52.490029 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.489453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52040)
0923 20:54:52.489702 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.489839 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 20:54:52.491222 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.490774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52048)
0923 20:54:52.490931 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.491047 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:54:52.534134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10953
I20260923 20:54:52.552076 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.551487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52056)
0923 20:54:52.551761 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.551879 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 20:54:52.553305 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.552866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52066)
0923 20:54:52.553040 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.553135 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 20:54:52.555173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 11087
Sep 23 20:54:52 dist-test-slave-plcg krb5kdc[10713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:54:52.572996 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.555905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.97.3:40073)
0923 20:54:52.556108 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:54:52.556114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:54:52.556137 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:54:52.556255 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:54:52.556260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:54:52.556350 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:54:52.556456 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:54:52.557664 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.558920 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.560136 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:54:52.560828 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:54:52.560954 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:54:52.565538 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:54:52.565552 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:54:52.565576 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:54:52.565637 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:54:52.569770 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:52.570550 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:52.570561 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:52.570565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:52.570754 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:54:52.571208 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:54:52.571218 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:54:52.571224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:54:52.571728 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:54:52.572056 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 20:54:52.572289 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 20:54:52.572539 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:54:52.574064 11210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:52.634486 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.633924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52074)
0923 20:54:52.634255 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.634336 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260923 20:54:52.635668 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.635190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52078)
0923 20:54:52.635430 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.635526 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260923 20:54:52.693383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 11087
I20260923 20:54:52.712620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 10726
I20260923 20:54:52.713726 10788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:54:52.822813 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:54:52.822254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33667 (local address 127.0.0.1:52086)
0923 20:54:52.822492 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 20:54:52.822599 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 20:54:52.822781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 10726
2026-09-23T20:54:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4657 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:54:52 dist-test-slave-plcg krb5kdc[11268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:54:52 dist-test-slave-plcg krb5kdc[11268](info): set up 2 sockets
Sep 23 20:54:52 dist-test-slave-plcg krb5kdc[11268](info): commencing operation
krb5kdc: starting...
W20260923 20:54:53.942775 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:54:54 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:54:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260923 20:54:54.283113 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskf1GrEz/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/postgres -l logfile start
2026-09-23 20:54:57.042 UTC [11298] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:54:57.042 UTC [11298] LOG: listening on IPv4 address "127.0.97.20", port 57249
2026-09-23 20:54:57.046 UTC [11298] LOG: listening on Unix socket "/tmp/.s.PGSQL.57249"
2026-09-23 20:54:57.071 UTC [11301] LOG: database system was shut down at 2026-09-23 20:54:55 UTC
2026-09-23 20:54:57.075 UTC [11298] LOG: database system is ready to accept connections
I20260923 20:54:58.054358 388 mini_postgres.cc:96] Postgres bound to 57249
2026-09-23 20:54:58.072 UTC [11309] FATAL: database "slave" does not exist
127.0.97.20:57249 - accepting connections
I20260923 20:54:58.073689 388 mini_ranger.cc:161] Starting Ranger...
I20260923 20:54:58.101509 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:54:58.637778 388 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:54:58.637957 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-admin
2026-09-23 20:54:58,946 [I] DB FLAVOR :POSTGRES
2026-09-23 20:54:58,946 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:54:58,946 [I] Checking connection..
2026-09-23 20:54:58,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:54:59,206 [I] Checking connection passed.
2026-09-23 20:54:59,206 [I] --------- Verifying version history table ---------
2026-09-23 20:54:59,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:54:59,460 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:54:59,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:54:59,706 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:54:59,706 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:54:59,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:54:59.980 UTC [11373] WARNING: there is no transaction in progress
2026-09-23 20:54:59,987 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:54:59,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:55:00,251 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:55:00,252 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:55:00,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:55:00,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:55:00,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-plcg.c.gcp-upstream.internal','N') ;"
2026-09-23 20:55:01,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:55:01,274 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:55:01,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:55:01,522 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:55:01,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:55:01,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:55:02,045 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:55:02,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:55:03.101 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.121 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.140 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.158 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.179 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.202 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.277 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.285 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.301 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.312 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.324 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.340 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.351 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.358 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.367 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:03.374 UTC [11481] WARNING: there is no transaction in progress
2026-09-23 20:55:04,112 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:55:04,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:55:04,358 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:55:04,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:55:04,620 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:55:04,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:55:04,867 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:55:04,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:55:05,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:55:05,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-plcg.c.gcp-upstream.internal';"
2026-09-23 20:55:05,650 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:55:05,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:55:05,919 [I] DB_PATCHES have already been applied
I20260923 20:55:05.925328 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskf1GrEz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:55:05.925448 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260923 20:55:05.941671 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:40659
Sep 23, 2026 8:55:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:55:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:55:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:55:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:55:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:55:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:55:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:55:07.113111 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.039s user 0.008s sys 0.035s
Sep 23, 2026 8:55:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:55:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 8:55:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:55:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:55:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:55:30.680310 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:55:30 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196930, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:55:30.878628 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:55:30.904088 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:55:31.845350 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:55:31.845538 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms
I20260923 20:55:31.845715 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:57249
2026-09-23 20:55:32,285 [I] DB FLAVOR :POSTGRES
2026-09-23 20:55:32,286 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:55:32,286 [I] Checking connection
2026-09-23 20:55:32,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:55:32,531 [I] connection success
2026-09-23 20:55:32,531 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:55:32,531 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:55:32,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:55:32,813 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:55:32,814 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:55:32,814 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:55:32,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:57249/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:55:33,126 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:55:33.670430 388 mini_ranger_kms.cc:310] Created kms service
I20260923 20:55:33.881754 388 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:55:33.936898 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:55:33 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196930, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:55:34.353128 388 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:55:34.353483 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:55:34.353547 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260923 20:55:34.374752 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11662
I20260923 20:55:34.374957 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:46217
Sep 23, 2026 8:55:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:55:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:55:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:55:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:55:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:55:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 20:55:36.664685 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.786s user 0.128s sys 0.336s
I20260923 20:55:36.664969 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:55:36.665182 388 mini_ranger_kms.cc:401] 127.0.97.20:46217/kms/v1/keys
Sep 23 20:55:36 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790196936, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 20:55:37 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196936, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 20:55:37 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196936, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 20:55:39 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:55:39.998672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37293
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37293
--ranger_config_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46217/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:55:40.368963 11712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:55:40.369267 11712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:55:40.369356 11712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:55:40.377257 11712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:55:40.377385 11712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:55:40.377431 11712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:55:40.377487 11712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:55:40.377535 11712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:55:40.388459 11712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:46217/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37293
--ranger_config_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:55:40.389758 11712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:55:40.391140 11712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:55:40.404181 11717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:40.404145 11718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:40.404179 11720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:55:40.405063 11712 server_base.cc:1034] running on GCE node
I20260923 20:55:40.405892 11712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:55:40.407344 11712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:55:40.408555 11712 hybrid_clock.cc:648] HybridClock initialized: now 1790196940408500 us; error 47 us; skew 500 ppm
I20260923 20:55:40.408888 11712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:55:40 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:55:40.412210 11712 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:55:40.413707 11712 webserver.cc:460] Webserver started at http://127.0.97.62:44437/ using document root <none> and password file <none>
I20260923 20:55:40.414453 11712 fs_manager.cc:359] Metadata directory not provided
I20260923 20:55:40.414556 11712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:55:40.414810 11712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:55:40 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:55:40.475941 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "daddc550da5445a5a904000a735f9cb4"
format_stamp: "Formatted at 2026-09-23 20:55:40 on dist-test-slave-plcg"
server_key: "49542d1ffeaa2f864411fc3476a2b1e8"
server_key_iv: "f138c83dc990aa7cadb68b17b1be54b4"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:40.476905 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "daddc550da5445a5a904000a735f9cb4"
format_stamp: "Formatted at 2026-09-23 20:55:40 on dist-test-slave-plcg"
server_key: "49542d1ffeaa2f864411fc3476a2b1e8"
server_key_iv: "f138c83dc990aa7cadb68b17b1be54b4"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:40.482648 11712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:55:40.502025 11727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:40.503834 11712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:55:40.504032 11712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "daddc550da5445a5a904000a735f9cb4"
format_stamp: "Formatted at 2026-09-23 20:55:40 on dist-test-slave-plcg"
server_key: "49542d1ffeaa2f864411fc3476a2b1e8"
server_key_iv: "f138c83dc990aa7cadb68b17b1be54b4"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:40.504302 11712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:55:40.533334 11712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:55:40.538811 11712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:55:40.539152 11712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:55:40.560326 11712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37293
I20260923 20:55:40.560379 11778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37293 every 8 connection(s)
I20260923 20:55:40.561802 11712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:55:40.566402 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:55:40.571640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 11712
I20260923 20:55:40.571874 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:55:40.572938 388 external_mini_cluster.cc:1428] Setting key 637e0735d48005ac6e3bd61e5c889bc2
I20260923 20:55:40.579511 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4: Bootstrap starting.
I20260923 20:55:40.582927 11779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4: Neither blocks nor log segments found. Creating new log.
I20260923 20:55:40.584033 11779 log.cc:824] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:55:40 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:55:40.587797 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4: No bootstrap required, opened a new log
I20260923 20:55:40.596143 11779 raft_consensus.cc:348] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daddc550da5445a5a904000a735f9cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37293 } }
I20260923 20:55:40.596449 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:40.575429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.0.97.62:37293)
0923 20:55:40.576286 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:40.576313 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:40.576356 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:55:40.578270 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:40.578297 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:40.578746 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:40.579173 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:40.580544 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:40.582049 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:40.583409 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:40.584105 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:40.584327 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:40.588634 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:40.588644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:40.588662 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:40.588700 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:40.592004 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:40.592830 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:40.592837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:40.592844 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:40.593012 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:40.593479 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:40.593486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:40.593494 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:40.594143 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:40.594482 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:40.594965 (+ 483us) server_negotiation.cc:299] Negotiation successful
0923 20:55:40.595403 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":363,"threads_started":1}
I20260923 20:55:40.596604 11779 raft_consensus.cc:374] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:55:40.596784 11779 raft_consensus.cc:729] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daddc550da5445a5a904000a735f9cb4, State: Initialized, Role: FOLLOWER
I20260923 20:55:40.597558 11779 consensus_queue.cc:260] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daddc550da5445a5a904000a735f9cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37293 } }
I20260923 20:55:40.597926 11779 raft_consensus.cc:388] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:55:40.598121 11779 raft_consensus.cc:482] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:55:40.598347 11779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:55:40.601341 11779 raft_consensus.cc:504] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daddc550da5445a5a904000a735f9cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37293 } }
I20260923 20:55:40.601943 11779 leader_election.cc:302] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daddc550da5445a5a904000a735f9cb4; no voters:
I20260923 20:55:40.602619 11779 leader_election.cc:288] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:55:40.602746 11784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:55:40.604012 11784 raft_consensus.cc:686] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [term 1 LEADER]: Becoming Leader. State: Replica: daddc550da5445a5a904000a735f9cb4, State: Running, Role: LEADER
I20260923 20:55:40.604619 11784 consensus_queue.cc:237] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daddc550da5445a5a904000a735f9cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37293 } }
I20260923 20:55:40.605145 11779 sys_catalog.cc:564] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:55:40.607522 11785 sys_catalog.cc:455] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daddc550da5445a5a904000a735f9cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daddc550da5445a5a904000a735f9cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37293 } } }
I20260923 20:55:40.607987 11785 sys_catalog.cc:458] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:55:40.608237 11786 sys_catalog.cc:455] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daddc550da5445a5a904000a735f9cb4. Latest consensus state: current_term: 1 leader_uuid: "daddc550da5445a5a904000a735f9cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daddc550da5445a5a904000a735f9cb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37293 } } }
I20260923 20:55:40.609365 11786 sys_catalog.cc:458] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:55:40.614504 11792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:55:40.619926 11792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:55:40.621147 11779 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:55:40.638237 11792 catalog_manager.cc:1347] Generated new cluster ID: bed6e59a5396414483690c835d2cef9e
I20260923 20:55:40.638425 11792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:55:40.665403 11792 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:55:40.667014 11792 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:55:40.686683 11792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4: Generated new TSK 0
I20260923 20:55:40.687685 11792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:55:41.396250 11821 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:55:41.510977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37293
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:55:41 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790196941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:55:41.993947 11824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:55:41.994269 11824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:55:41.994441 11824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:55:42.002064 11824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:55:42.002183 11824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:55:42.002380 11824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:55:42.020496 11824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:46217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:55:42.022444 11824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:55:42.024207 11824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:55:42.044126 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:42.051215 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:42.051903 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:55:42.052742 11824 server_base.cc:1034] running on GCE node
I20260923 20:55:42.053870 11824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:55:42.055145 11824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:55:42.056463 11824 hybrid_clock.cc:648] HybridClock initialized: now 1790196942056369 us; error 89 us; skew 500 ppm
I20260923 20:55:42.056933 11824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:55:42 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:55:42.065639 11824 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:55:42.067646 11824 webserver.cc:460] Webserver started at http://127.0.97.1:41143/ using document root <none> and password file <none>
I20260923 20:55:42.068491 11824 fs_manager.cc:359] Metadata directory not provided
I20260923 20:55:42.068642 11824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:55:42.069001 11824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:55:42 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:55:42.087355 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "e9507c4b22a243909a637de5903a659d"
format_stamp: "Formatted at 2026-09-23 20:55:42 on dist-test-slave-plcg"
server_key: "4a26737f4ee9e391b736ae845c9e8056"
server_key_iv: "3767ac90c993a863185eabc13790ba4d"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:42.088677 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e9507c4b22a243909a637de5903a659d"
format_stamp: "Formatted at 2026-09-23 20:55:42 on dist-test-slave-plcg"
server_key: "4a26737f4ee9e391b736ae845c9e8056"
server_key_iv: "3767ac90c993a863185eabc13790ba4d"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:42.107851 11824 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.012s
I20260923 20:55:42.135913 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:42.138052 11824 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.004s
I20260923 20:55:42.138386 11824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "e9507c4b22a243909a637de5903a659d"
format_stamp: "Formatted at 2026-09-23 20:55:42 on dist-test-slave-plcg"
server_key: "4a26737f4ee9e391b736ae845c9e8056"
server_key_iv: "3767ac90c993a863185eabc13790ba4d"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:42.138733 11824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:55:42.188678 11824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:55:42.195096 11824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:55:42.196358 11824 kserver.cc:163] Server-wide thread pool size limit: 3276
Sep 23 20:55:42 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:55:42.210381 11824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:55:42.219208 11824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:55:42.219408 11824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260923 20:55:42.219610 11824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:55:42.219882 11824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:42.345434 11824 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36479
I20260923 20:55:42.345490 11956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36479 every 8 connection(s)
I20260923 20:55:42.347342 11824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:55:42.352978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 11824
I20260923 20:55:42.353197 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:55:42.354107 388 external_mini_cluster.cc:1428] Setting key 600c595564c3c9bb9d1c84ae76b4aa7c
Sep 23 20:55:42 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:55:42.385335 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:42.351037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37293 (local address 127.0.97.1:35997)
0923 20:55:42.358523 (+ 7486us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:42.358581 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:42.361778 (+ 3197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:42.363919 (+ 2141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:42.363937 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:42.364697 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:55:42.366144 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:42.366189 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:42.367812 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:42.367818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:42.369186 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:42.369200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:42.370322 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:42.370332 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:42.370703 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:42.371575 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:55:42.371610 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:55:42.374844 (+ 3234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:55:42.379930 (+ 5086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:55:42.379945 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:55:42.379969 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:55:42.380709 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:55:42.381208 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:55:42.381216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:55:42.381223 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:55:42.381532 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:55:42.382360 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:55:42.382372 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:55:42.382856 (+ 484us) client_negotiation.cc:769] Sending connection context
0923 20:55:42.383158 (+ 302us) client_negotiation.cc:240] Negotiation successful
0923 20:55:42.383773 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7132,"thread_start_us":6865,"threads_started":1}
I20260923 20:55:42.386787 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:42.351663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35997 (local address 127.0.97.62:37293)
0923 20:55:42.363190 (+ 11527us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:42.363196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:42.363219 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:55:42.363287 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:42.363293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:42.363388 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:42.366401 (+ 3013us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:42.366587 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:42.367609 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:42.369373 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:42.369774 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:42.369992 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:42.375195 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:42.375205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:42.375212 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:42.375296 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:42.379699 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:42.380890 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:42.380898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:42.380905 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:42.381065 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:42.381694 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:42.381702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:42.381708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:42.382130 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:42.382278 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:42.385990 (+ 3712us) server_negotiation.cc:299] Negotiation successful
0923 20:55:42.386297 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11356,"thread_start_us":11222,"threads_started":1}
I20260923 20:55:42.388468 11957 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37293
I20260923 20:55:42.388837 11957 heartbeater.cc:464] Registering TS with master...
I20260923 20:55:42.389804 11957 heartbeater.cc:511] Master 127.0.97.62:37293 requested a full tablet report, sending...
I20260923 20:55:42.392716 11744 ts_manager.cc:194] Registered new tserver with Master: e9507c4b22a243909a637de5903a659d (127.0.97.1:36479)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:55:42.394893 11744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:55:42.433018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37293
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:55:42 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 20:55:42.966040 11963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:55:42.966590 11963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:55:42.966799 11963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:55:42.976918 11963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:55:42.977523 11963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:55:42.977730 11963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:55:42.988533 11963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:46217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:55:42.990044 11963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:55:42.991397 11963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:55:43.012037 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:43.013235 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:43.022248 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:55:43.022939 11963 server_base.cc:1034] running on GCE node
I20260923 20:55:43.023720 11963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:55:43.024938 11963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:55:43.026238 11963 hybrid_clock.cc:648] HybridClock initialized: now 1790196943026167 us; error 71 us; skew 500 ppm
I20260923 20:55:43.026636 11963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:55:43 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:55:43.030238 11963 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:55:43.035386 11963 webserver.cc:460] Webserver started at http://127.0.97.2:39197/ using document root <none> and password file <none>
I20260923 20:55:43.036284 11963 fs_manager.cc:359] Metadata directory not provided
I20260923 20:55:43.036502 11963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:55:43.036859 11963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:55:43 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:55:43.060015 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "f6c57b146831456aafecd2a426c4cee2"
format_stamp: "Formatted at 2026-09-23 20:55:43 on dist-test-slave-plcg"
server_key: "31e40e025b92fd269cab92ae208be20a"
server_key_iv: "a8df5ea00dedf5e98847b9aac671d021"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:43.062729 11963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f6c57b146831456aafecd2a426c4cee2"
format_stamp: "Formatted at 2026-09-23 20:55:43 on dist-test-slave-plcg"
server_key: "31e40e025b92fd269cab92ae208be20a"
server_key_iv: "a8df5ea00dedf5e98847b9aac671d021"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:43.083585 11963 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.012s sys 0.004s
I20260923 20:55:43.114542 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:43.116844 11963 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.007s sys 0.004s
I20260923 20:55:43.117177 11963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "f6c57b146831456aafecd2a426c4cee2"
format_stamp: "Formatted at 2026-09-23 20:55:43 on dist-test-slave-plcg"
server_key: "31e40e025b92fd269cab92ae208be20a"
server_key_iv: "a8df5ea00dedf5e98847b9aac671d021"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:43.117529 11963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:55:43.164687 11963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:55:43.170253 11963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:55:43.170651 11963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:55:43.178507 11963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:55:43.187932 11963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:55:43.188246 11963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:55:43.188424 11963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:55:43.188513 11963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:43.399679 11957 heartbeater.cc:503] Master 127.0.97.62:37293 was elected leader, sending a full tablet report...
I20260923 20:55:43.530933 11963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43697
I20260923 20:55:43.530958 12099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43697 every 8 connection(s)
I20260923 20:55:43.532639 11963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:55:43.534482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 11963
I20260923 20:55:43.534653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:55:43.535158 388 external_mini_cluster.cc:1428] Setting key 1bce242871b8d70cb681b8840aa1c820
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 23 20:55:43 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:55:43.591150 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:43.536973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57363 (local address 127.0.97.62:37293)
0923 20:55:43.558946 (+ 21973us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:43.558953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:43.560137 (+ 1184us) server_negotiation.cc:407] Connection header received
0923 20:55:43.564327 (+ 4190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:43.564333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:43.564425 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:43.564508 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:43.572816 (+ 8308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:43.574003 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:43.576900 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:43.577535 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:43.577649 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:43.583050 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:43.583057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:43.583061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:43.583111 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:43.586633 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:43.587790 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:43.587799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:43.587805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:43.587975 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:43.588614 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:43.588622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:43.588628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:43.589041 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:43.589181 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:43.590317 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0923 20:55:43.590605 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21798,"thread_start_us":21997,"threads_started":1}
I20260923 20:55:43.593875 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:43.536218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37293 (local address 127.0.97.2:57363)
0923 20:55:43.559733 (+ 23515us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:43.559792 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:43.563615 (+ 3823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:43.565079 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:43.565092 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:43.565724 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:55:43.566589 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:43.566616 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:43.574287 (+ 7671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:43.574292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:43.576660 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:43.576694 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:43.578063 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:43.578070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:43.578396 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:43.579321 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:55:43.579355 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:55:43.582689 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:55:43.586847 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:55:43.586862 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:55:43.586888 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:55:43.587617 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:55:43.588141 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:55:43.588149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:55:43.588156 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:55:43.588464 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:55:43.589290 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:55:43.589303 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:55:43.589806 (+ 503us) client_negotiation.cc:769] Sending connection context
0923 20:55:43.591624 (+ 1818us) client_negotiation.cc:240] Negotiation successful
0923 20:55:43.592320 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23123,"thread_start_us":24449,"threads_started":1}
I20260923 20:55:43.595889 12100 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37293
I20260923 20:55:43.596218 12100 heartbeater.cc:464] Registering TS with master...
I20260923 20:55:43.597137 12100 heartbeater.cc:511] Master 127.0.97.62:37293 requested a full tablet report, sending...
I20260923 20:55:43.598892 11744 ts_manager.cc:194] Registered new tserver with Master: f6c57b146831456aafecd2a426c4cee2 (127.0.97.2:43697)
I20260923 20:55:43.600164 11744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:55:43.617926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37293
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:55:43.977636 12106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:55:43.977981 12106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:55:43.978084 12106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:55:43.985667 12106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:55:43.985793 12106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:55:43.985935 12106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:55:43.996002 12106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:46217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:55:43.997332 12106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:55:43.998709 12106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:55:44.013237 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:44.013262 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:55:44.013371 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:55:44.014216 12106 server_base.cc:1034] running on GCE node
I20260923 20:55:44.014967 12106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:55:44.016393 12106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:55:44.017609 12106 hybrid_clock.cc:648] HybridClock initialized: now 1790196944017530 us; error 63 us; skew 500 ppm
I20260923 20:55:44.018030 12106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:55:44 dist-test-slave-plcg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:55:44.021299 12106 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:55:44.022835 12106 webserver.cc:460] Webserver started at http://127.0.97.3:38845/ using document root <none> and password file <none>
I20260923 20:55:44.023643 12106 fs_manager.cc:359] Metadata directory not provided
I20260923 20:55:44.023797 12106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:55:44.024128 12106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:55:44 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:55:44.039812 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "efe28e7804664b1fbe81c18a72f60bd6"
format_stamp: "Formatted at 2026-09-23 20:55:44 on dist-test-slave-plcg"
server_key: "36f5bc6edbce1da7972c271c88c2c02c"
server_key_iv: "d0e8b9edc9dc15eb8e149b9ac5c95be2"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:44.040841 12106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "efe28e7804664b1fbe81c18a72f60bd6"
format_stamp: "Formatted at 2026-09-23 20:55:44 on dist-test-slave-plcg"
server_key: "36f5bc6edbce1da7972c271c88c2c02c"
server_key_iv: "d0e8b9edc9dc15eb8e149b9ac5c95be2"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:44.046718 12106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:55:44.064972 12121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:44.066648 12106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:55:44.066857 12106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "efe28e7804664b1fbe81c18a72f60bd6"
format_stamp: "Formatted at 2026-09-23 20:55:44 on dist-test-slave-plcg"
server_key: "36f5bc6edbce1da7972c271c88c2c02c"
server_key_iv: "d0e8b9edc9dc15eb8e149b9ac5c95be2"
server_key_version: "kuduclusterkey@0"
I20260923 20:55:44.067091 12106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:55:44.079849 12106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:55:44.083994 12106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:55:44.084316 12106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:55:44.085314 12106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:55:44.087292 12106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:55:44.087404 12106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:44.087550 12106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:55:44.087620 12106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:55:44.126880 12106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32909
I20260923 20:55:44.126914 12233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32909 every 8 connection(s)
I20260923 20:55:44.128247 12106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:55:44.132722 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 12106
I20260923 20:55:44.132902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:55:44.133481 388 external_mini_cluster.cc:1428] Setting key 1cdf9644f1e4378dbd060d36a2e8ea06
Sep 23 20:55:44 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:55:44.151012 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.131355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43131 (local address 127.0.97.62:37293)
0923 20:55:44.131871 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.131878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.133238 (+ 1360us) server_negotiation.cc:407] Connection header received
0923 20:55:44.135832 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.135839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.135930 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.136048 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:44.137762 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.138605 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.140723 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.141165 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.141278 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.144823 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:44.144830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:44.144834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:44.144881 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:44.147493 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.148511 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.148516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.148518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.148696 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.149207 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.149226 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.149230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.149587 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:44.149712 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.150547 (+ 835us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.150738 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":235,"threads_started":1}
I20260923 20:55:44.151950 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.132053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37293 (local address 127.0.97.3:43131)
0923 20:55:44.132997 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:44.133058 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:44.135454 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:44.136270 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:44.136279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:44.136910 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:55:44.137574 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.137594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.138788 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.138792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.140542 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.140563 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.141395 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.141400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.141615 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.142416 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:55:44.142441 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:55:44.144541 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:55:44.147690 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:55:44.147702 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:55:44.147723 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:55:44.148377 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:55:44.148818 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:55:44.148823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:55:44.148827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:55:44.149095 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:55:44.149731 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:55:44.149740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:55:44.150144 (+ 404us) client_negotiation.cc:769] Sending connection context
0923 20:55:44.150415 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 20:55:44.150823 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":384,"threads_started":1}
I20260923 20:55:44.153613 12234 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37293
I20260923 20:55:44.153967 12234 heartbeater.cc:464] Registering TS with master...
I20260923 20:55:44.154664 12234 heartbeater.cc:511] Master 127.0.97.62:37293 requested a full tablet report, sending...
I20260923 20:55:44.156092 11744 ts_manager.cc:194] Registered new tserver with Master: efe28e7804664b1fbe81c18a72f60bd6 (127.0.97.3:32909)
I20260923 20:55:44.156916 11744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43131
I20260923 20:55:44.161947 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:55:44.180019 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.166806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.0.97.62:37293)
0923 20:55:44.166989 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.166994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.167390 (+ 396us) server_negotiation.cc:407] Connection header received
0923 20:55:44.167525 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.167529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.167611 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.167701 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:44.169576 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.170405 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.171840 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.172340 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.172440 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.174885 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:44.174890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:44.174893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:44.174927 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:44.177609 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.178283 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.178287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.178290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.178417 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.178753 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.178757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.178759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.179036 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:44.179211 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.179560 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.179776 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:55:44.184221 11744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:55:44.191789 11744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:55:44.230858 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.205692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36479 (local address 127.0.0.1:49348)
0923 20:55:44.218018 (+ 12326us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:44.218042 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:44.218474 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:44.222616 (+ 4142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:44.222627 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:44.222670 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:55:44.223608 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.223627 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.225485 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.225491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.227887 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.227910 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.229659 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.229666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.229960 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.230036 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 20:55:44.230180 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 20:55:44.230385 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12159,"mutex_wait_us":54,"thread_start_us":9991,"threads_started":1}
I20260923 20:55:44.232127 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.206782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.0.97.1:36479)
0923 20:55:44.221536 (+ 14754us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.221543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.221570 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:55:44.221956 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.221968 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.222299 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.222785 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:55:44.223821 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.225323 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.228066 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.229416 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.229564 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.229843 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.231580 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.231785 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14590,"thread_start_us":15124,"threads_started":1}
I20260923 20:55:44.232285 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.205227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32909 (local address 127.0.0.1:33172)
0923 20:55:44.216123 (+ 10896us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:44.216173 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:44.216404 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:44.224967 (+ 8563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:44.224972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:44.225000 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:55:44.225385 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.225400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.227234 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.227242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.229137 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.229148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.230982 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.230989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.231652 (+ 663us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.231694 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:55:44.231771 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:55:44.231924 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10627,"thread_start_us":17055,"threads_started":1}
I20260923 20:55:44.233197 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.205580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.0.97.3:32909)
0923 20:55:44.224414 (+ 18834us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.224422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.224459 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:55:44.224560 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.224565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.224763 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.224939 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:55:44.225566 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.227070 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.229302 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.230736 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.232165 (+ 1429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.232619 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.232701 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.232854 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18638,"thread_start_us":18568,"threads_started":1}
I20260923 20:55:44.233438 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.206217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.0.97.2:43697)
0923 20:55:44.225338 (+ 19121us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.225343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.225364 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:55:44.225479 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.225485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.225717 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.225961 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:55:44.226727 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.228506 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.230860 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.232290 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.232416 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.232571 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.232990 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.233152 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18995,"thread_start_us":17501,"threads_started":1}
I20260923 20:55:44.233440 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.205953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43697 (local address 127.0.0.1:45550)
0923 20:55:44.223844 (+ 17891us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:44.223864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:44.223975 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:44.225936 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:44.225942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:44.225972 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:55:44.226511 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.226523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.228672 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.228677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.230665 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.230679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.232691 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.232696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.232805 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.232826 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:55:44.232941 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:55:44.233071 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16939,"mutex_wait_us":123,"thread_start_us":12179,"threads_started":1}
I20260923 20:55:44.236608 11892 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa5bc0f6717406fb3790b3bf6d8853e (DEFAULT_TABLE table=test-table [id=193b8bffaaf2455e9d408dbd66006a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:55:44.236862 12035 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa5bc0f6717406fb3790b3bf6d8853e (DEFAULT_TABLE table=test-table [id=193b8bffaaf2455e9d408dbd66006a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:55:44.237035 12169 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa5bc0f6717406fb3790b3bf6d8853e (DEFAULT_TABLE table=test-table [id=193b8bffaaf2455e9d408dbd66006a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:55:44.238127 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa5bc0f6717406fb3790b3bf6d8853e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:55:44.238245 11892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa5bc0f6717406fb3790b3bf6d8853e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:55:44.238348 12169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa5bc0f6717406fb3790b3bf6d8853e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:55:44.251151 12253 tablet_bootstrap.cc:492] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Bootstrap starting.
I20260923 20:55:44.253121 12254 tablet_bootstrap.cc:492] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Bootstrap starting.
I20260923 20:55:44.254724 12255 tablet_bootstrap.cc:492] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Bootstrap starting.
I20260923 20:55:44.256933 12253 tablet_bootstrap.cc:654] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Neither blocks nor log segments found. Creating new log.
I20260923 20:55:44.256933 12254 tablet_bootstrap.cc:654] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Neither blocks nor log segments found. Creating new log.
I20260923 20:55:44.257679 12255 tablet_bootstrap.cc:654] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Neither blocks nor log segments found. Creating new log.
I20260923 20:55:44.258687 12255 log.cc:824] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:55:44.258728 12253 log.cc:824] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:55:44.258728 12254 log.cc:824] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:55:44.261747 12255 tablet_bootstrap.cc:492] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: No bootstrap required, opened a new log
I20260923 20:55:44.262084 12255 ts_tablet_manager.cc:1397] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 20:55:44.262099 12254 tablet_bootstrap.cc:492] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: No bootstrap required, opened a new log
I20260923 20:55:44.262077 12253 tablet_bootstrap.cc:492] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: No bootstrap required, opened a new log
I20260923 20:55:44.262372 12253 ts_tablet_manager.cc:1397] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 20:55:44.262372 12254 ts_tablet_manager.cc:1397] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:55:44.270241 12255 raft_consensus.cc:348] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.270674 12255 raft_consensus.cc:374] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:55:44.270774 12255 raft_consensus.cc:729] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe28e7804664b1fbe81c18a72f60bd6, State: Initialized, Role: FOLLOWER
I20260923 20:55:44.271559 12255 consensus_queue.cc:260] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.273663 12255 ts_tablet_manager.cc:1428] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Time spent starting tablet: real 0.011s user 0.002s sys 0.009s
I20260923 20:55:44.274211 12234 heartbeater.cc:503] Master 127.0.97.62:37293 was elected leader, sending a full tablet report...
I20260923 20:55:44.274317 12253 raft_consensus.cc:348] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.274315 12254 raft_consensus.cc:348] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.274812 12254 raft_consensus.cc:374] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:55:44.274812 12253 raft_consensus.cc:374] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:55:44.274925 12253 raft_consensus.cc:729] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c57b146831456aafecd2a426c4cee2, State: Initialized, Role: FOLLOWER
I20260923 20:55:44.274921 12254 raft_consensus.cc:729] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9507c4b22a243909a637de5903a659d, State: Initialized, Role: FOLLOWER
I20260923 20:55:44.275604 12253 consensus_queue.cc:260] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.275729 12254 consensus_queue.cc:260] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.280967 12100 heartbeater.cc:503] Master 127.0.97.62:37293 was elected leader, sending a full tablet report...
I20260923 20:55:44.282203 12253 ts_tablet_manager.cc:1428] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
I20260923 20:55:44.282857 12259 raft_consensus.cc:482] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:55:44.283509 12259 raft_consensus.cc:504] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.285993 12254 ts_tablet_manager.cc:1428] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Time spent starting tablet: real 0.024s user 0.016s sys 0.008s
I20260923 20:55:44.287021 12259 leader_election.cc:288] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9507c4b22a243909a637de5903a659d (127.0.97.1:36479), f6c57b146831456aafecd2a426c4cee2 (127.0.97.2:43697)
I20260923 20:55:44.294322 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.285809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36479 (local address 127.0.97.3:59339)
0923 20:55:44.286750 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:44.286770 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:44.286923 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:44.287968 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:44.287973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:44.288005 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:55:44.288193 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.288207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.289865 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.289874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.291832 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.291847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.293610 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.293614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.293799 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.293903 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 20:55:44.293983 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 20:55:44.294082 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":2}
I20260923 20:55:44.294464 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.286173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59339 (local address 127.0.97.1:36479)
0923 20:55:44.286566 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.286574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.286872 (+ 298us) server_negotiation.cc:407] Connection header received
0923 20:55:44.287120 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.287125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.287204 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.287291 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:55:44.288368 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.289434 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.292019 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.293432 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.293631 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.293760 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.294040 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.294179 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260923 20:55:44.295559 11912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa5bc0f6717406fb3790b3bf6d8853e" candidate_uuid: "efe28e7804664b1fbe81c18a72f60bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9507c4b22a243909a637de5903a659d" is_pre_election: true
I20260923 20:55:44.296202 11912 raft_consensus.cc:2454] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe28e7804664b1fbe81c18a72f60bd6 in term 0.
I20260923 20:55:44.296909 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.287494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58401 (local address 127.0.97.2:43697)
0923 20:55:44.287744 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.287753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.289919 (+ 2166us) server_negotiation.cc:407] Connection header received
0923 20:55:44.290196 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.290202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.290285 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.290376 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:55:44.290994 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.291987 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.294764 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.295910 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.296041 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.296132 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.296498 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.296658 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:55:44.297003 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.287369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43697 (local address 127.0.97.3:58401)
0923 20:55:44.289788 (+ 2419us) negotiation.cc:107] Waiting for socket to connect
0923 20:55:44.289839 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:55:44.289983 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:55:44.290420 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:55:44.290424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:55:44.290455 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:55:44.290816 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.290824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.292167 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.292173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.294300 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:55:44.294318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.296102 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.296119 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:55:44.296256 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.296278 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:55:44.296469 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 20:55:44.296607 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1832,"mutex_wait_us":30,"thread_start_us":1703,"threads_started":1}
I20260923 20:55:44.296962 12122 leader_election.cc:302] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9507c4b22a243909a637de5903a659d, efe28e7804664b1fbe81c18a72f60bd6; no voters:
I20260923 20:55:44.297454 12259 raft_consensus.cc:2781] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:55:44.297626 12259 raft_consensus.cc:482] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:55:44.297734 12259 raft_consensus.cc:3037] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:55:44.298225 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa5bc0f6717406fb3790b3bf6d8853e" candidate_uuid: "efe28e7804664b1fbe81c18a72f60bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c57b146831456aafecd2a426c4cee2" is_pre_election: true
I20260923 20:55:44.298718 12055 raft_consensus.cc:2454] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe28e7804664b1fbe81c18a72f60bd6 in term 0.
I20260923 20:55:44.300431 12259 raft_consensus.cc:504] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.301034 12259 leader_election.cc:288] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [CANDIDATE]: Term 1 election: Requested vote from peers e9507c4b22a243909a637de5903a659d (127.0.97.1:36479), f6c57b146831456aafecd2a426c4cee2 (127.0.97.2:43697)
I20260923 20:55:44.301574 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa5bc0f6717406fb3790b3bf6d8853e" candidate_uuid: "efe28e7804664b1fbe81c18a72f60bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c57b146831456aafecd2a426c4cee2"
I20260923 20:55:44.301810 12055 raft_consensus.cc:3037] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:55:44.301926 11912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa5bc0f6717406fb3790b3bf6d8853e" candidate_uuid: "efe28e7804664b1fbe81c18a72f60bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9507c4b22a243909a637de5903a659d"
I20260923 20:55:44.302161 11912 raft_consensus.cc:3037] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:55:44.304404 12055 raft_consensus.cc:2454] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe28e7804664b1fbe81c18a72f60bd6 in term 1.
I20260923 20:55:44.304879 11912 raft_consensus.cc:2454] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe28e7804664b1fbe81c18a72f60bd6 in term 1.
I20260923 20:55:44.304893 12124 leader_election.cc:302] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efe28e7804664b1fbe81c18a72f60bd6, f6c57b146831456aafecd2a426c4cee2; no voters:
I20260923 20:55:44.305264 12259 raft_consensus.cc:2781] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:55:44.305428 12259 raft_consensus.cc:686] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 1 LEADER]: Becoming Leader. State: Replica: efe28e7804664b1fbe81c18a72f60bd6, State: Running, Role: LEADER
I20260923 20:55:44.306053 12259 consensus_queue.cc:237] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } }
I20260923 20:55:44.311105 11744 catalog_manager.cc:5495] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to efe28e7804664b1fbe81c18a72f60bd6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "efe28e7804664b1fbe81c18a72f60bd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe28e7804664b1fbe81c18a72f60bd6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 } health_report { overall_health: UNKNOWN } } }
W20260923 20:55:44.314954 12101 tablet.cc:2369] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:55:44.339236 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.333309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.0.97.3:32909)
0923 20:55:44.333507 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.333512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.333535 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:55:44.333748 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.333752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.333861 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.333996 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:55:44.334436 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.335211 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.336750 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.337168 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.337284 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.337957 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:55:44.338533 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:55:44.338771 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.338945 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.339022 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:55:44.349005 11912 raft_consensus.cc:1261] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 1 FOLLOWER]: Refusing update from remote peer efe28e7804664b1fbe81c18a72f60bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:55:44.349030 12055 raft_consensus.cc:1261] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 1 FOLLOWER]: Refusing update from remote peer efe28e7804664b1fbe81c18a72f60bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:55:44.350162 12260 consensus_queue.cc:1035] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9507c4b22a243909a637de5903a659d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:55:44.350587 12259 consensus_queue.cc:1035] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c57b146831456aafecd2a426c4cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:55:44.356490 11958 tablet.cc:2369] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:55:44.366022 12266 mvcc.cc:205] Tried to move back new op lower bound from 7332646684041515008 to 7332646683881291776. Current Snapshot: MvccSnapshot[applied={T|T < 7332646684041515008}]
I20260923 20:55:44.383106 11744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:55:44.383416 11744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:55:44.387523 11744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daddc550da5445a5a904000a735f9cb4: Sending DeleteTablet for 3 replicas of tablet 5fa5bc0f6717406fb3790b3bf6d8853e
I20260923 20:55:44.388844 12169 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa5bc0f6717406fb3790b3bf6d8853e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:55:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33172
I20260923 20:55:44.388902 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa5bc0f6717406fb3790b3bf6d8853e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:55:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45550
I20260923 20:55:44.389698 11892 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa5bc0f6717406fb3790b3bf6d8853e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:55:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49348
I20260923 20:55:44.390084 12276 tablet_replica.cc:330] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: stopping tablet replica
I20260923 20:55:44.390566 12276 raft_consensus.cc:2229] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:55:44.391140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 11824
I20260923 20:55:44.391273 12276 raft_consensus.cc:2258] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:55:44.395155 12276 ts_tablet_manager.cc:1905] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:55:44.403234 12276 ts_tablet_manager.cc:1918] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:55:44.403429 12276 log.cc:1196] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/5fa5bc0f6717406fb3790b3bf6d8853e
I20260923 20:55:44.404074 12276 ts_tablet_manager.cc:1939] T 5fa5bc0f6717406fb3790b3bf6d8853e P efe28e7804664b1fbe81c18a72f60bd6: Deleting consensus metadata
I20260923 20:55:44.406154 11731 catalog_manager.cc:4841] TS efe28e7804664b1fbe81c18a72f60bd6 (127.0.97.3:32909): tablet 5fa5bc0f6717406fb3790b3bf6d8853e (table test-table [id=193b8bffaaf2455e9d408dbd66006a64]) successfully deleted
I20260923 20:55:44.407387 12277 tablet_replica.cc:330] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: stopping tablet replica
I20260923 20:55:44.407907 12277 raft_consensus.cc:2229] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:55:44.409072 12277 raft_consensus.cc:2258] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:55:44.410414 12275 tablet_replica.cc:330] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: stopping tablet replica
I20260923 20:55:44.411121 12275 raft_consensus.cc:2229] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:55:44.411655 12275 raft_consensus.cc:2258] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:55:44.412179 12277 ts_tablet_manager.cc:1905] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:55:44.413954 12275 ts_tablet_manager.cc:1905] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:55:44 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:55:44.420135 12277 ts_tablet_manager.cc:1918] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:55:44.420336 12277 log.cc:1196] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/5fa5bc0f6717406fb3790b3bf6d8853e
I20260923 20:55:44.420951 12277 ts_tablet_manager.cc:1939] T 5fa5bc0f6717406fb3790b3bf6d8853e P e9507c4b22a243909a637de5903a659d: Deleting consensus metadata
I20260923 20:55:44.422016 12275 ts_tablet_manager.cc:1918] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:55:44.422215 12275 log.cc:1196] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/5fa5bc0f6717406fb3790b3bf6d8853e
I20260923 20:55:44.422756 12275 ts_tablet_manager.cc:1939] T 5fa5bc0f6717406fb3790b3bf6d8853e P f6c57b146831456aafecd2a426c4cee2: Deleting consensus metadata
I20260923 20:55:44.423211 11728 catalog_manager.cc:4841] TS e9507c4b22a243909a637de5903a659d (127.0.97.1:36479): tablet 5fa5bc0f6717406fb3790b3bf6d8853e (table test-table [id=193b8bffaaf2455e9d408dbd66006a64]) successfully deleted
I20260923 20:55:44.424813 11730 catalog_manager.cc:4841] TS f6c57b146831456aafecd2a426c4cee2 (127.0.97.2:43697): tablet 5fa5bc0f6717406fb3790b3bf6d8853e (table test-table [id=193b8bffaaf2455e9d408dbd66006a64]) successfully deleted
I20260923 20:55:44.426446 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.392393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.0.97.1:36479)
0923 20:55:44.392605 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.392613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.405260 (+ 12647us) server_negotiation.cc:407] Connection header received
0923 20:55:44.405835 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.405843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.405983 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.406075 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:44.408383 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.410155 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.412102 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.412728 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.412845 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.418630 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:44.418647 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:44.418670 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:44.418721 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:44.422869 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.423982 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.423990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.423993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.424134 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.424663 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.424669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.424672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.425173 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:44.425496 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.425771 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.426043 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:55:44.427166 11952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:55:44.521062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 11824
I20260923 20:55:44.544171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 11963
Sep 23 20:55:44 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:55:44.566437 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.545040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.0.97.2:43697)
0923 20:55:44.545225 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.545233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.545256 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:55:44.545504 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.545511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.545628 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.545754 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:44.547519 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.548689 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.550776 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.551499 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.551652 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.557505 (+ 5853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:44.557519 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:44.557535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:44.557596 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:44.562281 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.563413 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.563424 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.563428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.563628 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.564132 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.564140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.564146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.564667 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:44.564973 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.565561 (+ 588us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.565842 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:55:44.567166 12095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:55:44.666388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 11963
I20260923 20:55:44.683930 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 12106
Sep 23 20:55:44 dist-test-slave-plcg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:55:44.701488 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:55:44.684971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.0.97.3:32909)
0923 20:55:44.685170 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:55:44.685178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:55:44.685193 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:55:44.685248 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:55:44.685253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:55:44.685313 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:55:44.685429 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:55:44.686849 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.687667 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.689027 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:55:44.689503 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:55:44.689656 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:55:44.694711 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:55:44.694732 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:55:44.694752 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:55:44.694816 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:55:44.698180 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.699047 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.699056 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.699058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.699232 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:55:44.699650 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:55:44.699656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:55:44.699659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:55:44.700047 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:55:44.700388 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 20:55:44.700774 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 20:55:44.701078 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:55:44.702368 12229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:55:44.830835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 12106
I20260923 20:55:44.850670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 11712
I20260923 20:55:44.851899 11774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:55:44.967790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 11712
I20260923 20:55:44.990128 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:55:45.326380 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:55:45.326612 388 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:55:45.672937 388 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:55:45.673 UTC [11298] LOG: received smart shutdown request
2026-09-23 20:55:45.676 UTC [11298] LOG: background worker "logical replication launcher" (PID 11307) exited with exit code 1
2026-09-23 20:55:45.677 UTC [11302] LOG: shutting down
2026-09-23 20:55:45.870 UTC [11298] LOG: database system is shut down
2026-09-23T20:55:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53179 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:55:46 dist-test-slave-plcg krb5kdc[12291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:55:46 dist-test-slave-plcg krb5kdc[12291](info): set up 2 sockets
Sep 23 20:55:46 dist-test-slave-plcg krb5kdc[12291](info): commencing operation
krb5kdc: starting...
W20260923 20:55:47.118211 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.011s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:55:47 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:55:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260923 20:55:47.461174 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskf1GrEz/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/postgres -l logfile start
2026-09-23 20:55:50.060 UTC [12321] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:55:50.060 UTC [12321] LOG: listening on IPv4 address "127.0.97.20", port 41709
2026-09-23 20:55:50.064 UTC [12321] LOG: listening on Unix socket "/tmp/.s.PGSQL.41709"
2026-09-23 20:55:50.086 UTC [12324] LOG: database system was shut down at 2026-09-23 20:55:48 UTC
2026-09-23 20:55:50.090 UTC [12321] LOG: database system is ready to accept connections
I20260923 20:55:51.069792 388 mini_postgres.cc:96] Postgres bound to 41709
2026-09-23 20:55:51.087 UTC [12332] FATAL: database "slave" does not exist
127.0.97.20:41709 - accepting connections
I20260923 20:55:51.087952 388 mini_ranger.cc:161] Starting Ranger...
I20260923 20:55:51.115566 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:55:51.645675 388 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:55:51.645803 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-admin
2026-09-23 20:55:51,941 [I] DB FLAVOR :POSTGRES
2026-09-23 20:55:51,942 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:55:51,942 [I] Checking connection..
2026-09-23 20:55:51,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:55:52,202 [I] Checking connection passed.
2026-09-23 20:55:52,203 [I] --------- Verifying version history table ---------
2026-09-23 20:55:52,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:55:52,480 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:55:52,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:55:52,757 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:55:52,757 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:55:52,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:55:53.020 UTC [12396] WARNING: there is no transaction in progress
2026-09-23 20:55:53,027 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:55:53,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:55:53,295 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:55:53,295 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:55:53,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:55:53,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:55:53,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-plcg.c.gcp-upstream.internal','N') ;"
2026-09-23 20:55:54,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:55:54,295 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:55:54,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:55:54,536 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:55:54,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:55:54,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:55:55,036 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:55:55,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:55:56.048 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.066 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.083 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.101 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.117 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.137 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.197 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.204 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.220 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.230 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.241 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.257 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.268 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.275 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.283 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:56.290 UTC [12504] WARNING: there is no transaction in progress
2026-09-23 20:55:57,013 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:55:57,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:55:57,271 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:55:57,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:55:57,544 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:55:57,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:55:57,818 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:55:57,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:55:58,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:55:58,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-plcg.c.gcp-upstream.internal';"
2026-09-23 20:55:58,569 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:55:58,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:55:58,833 [I] DB_PATCHES have already been applied
I20260923 20:55:58.839478 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskf1GrEz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:55:58.839591 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260923 20:55:58.854717 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:57093
Sep 23, 2026 8:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:55:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:56:00.008064 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.920s user 0.003s sys 0.040s
Sep 23, 2026 8:56:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:56:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 8:56:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:56:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:56:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:56:23.958020 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:56:24 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196984, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:56:24.126937 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:56:24.149241 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:56:25.231299 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:56:25.231444 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms
I20260923 20:56:25.231523 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:41709
2026-09-23 20:56:25,598 [I] DB FLAVOR :POSTGRES
2026-09-23 20:56:25,599 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:56:25,599 [I] Checking connection
2026-09-23 20:56:25,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:56:25,875 [I] connection success
2026-09-23 20:56:25,875 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:56:25,875 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:56:25,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:56:26,145 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:56:26,146 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:56:26,146 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:56:26,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41709/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:56:26,414 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:56:26.969271 388 mini_ranger_kms.cc:310] Created kms service
I20260923 20:56:27.201145 388 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:56:27.254567 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:56:27 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196984, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:27.672683 388 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:56:27.673040 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskf1GrEz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskf1GrEz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:56:27.673096 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260923 20:56:27.690713 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12685
I20260923 20:56:27.690860 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:60455
Sep 23, 2026 8:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:56:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:56:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 20:56:29 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790196989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:30.004382 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.877s user 0.053s sys 0.328s
I20260923 20:56:30.004621 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:56:30.004802 388 mini_ranger_kms.cc:401] 127.0.97.20:60455/kms/v1/keys
Sep 23 20:56:30 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 20:56:30 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 20:56:33 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:56:33.222209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44171
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44171
--ranger_config_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:33.574204 12735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:33.574519 12735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:33.574672 12735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:33.582919 12735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:33.583070 12735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:33.583170 12735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:33.583249 12735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:33.583309 12735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:33.594470 12735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:60455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44171
--ranger_config_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:33.595887 12735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:33.597415 12735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:33.610545 12741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:33.610495 12743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:33.610617 12740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:33.611340 12735 server_base.cc:1034] running on GCE node
I20260923 20:56:33.612177 12735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:33.613792 12735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:33.615113 12735 hybrid_clock.cc:648] HybridClock initialized: now 1790196993615061 us; error 46 us; skew 500 ppm
I20260923 20:56:33.615533 12735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:33 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:33.618901 12735 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:56:33.620630 12735 webserver.cc:460] Webserver started at http://127.0.97.62:45755/ using document root <none> and password file <none>
I20260923 20:56:33.621320 12735 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:33.621429 12735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:33.621722 12735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:56:33 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:33.681223 12735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "70afcfffeb2d4eb0b5c460f544136625"
format_stamp: "Formatted at 2026-09-23 20:56:33 on dist-test-slave-plcg"
server_key: "430bdcdcc76a16964007b8188461a40a"
server_key_iv: "66ff88a1c535b8dc7fbc49e561372061"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:33.682219 12735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "70afcfffeb2d4eb0b5c460f544136625"
format_stamp: "Formatted at 2026-09-23 20:56:33 on dist-test-slave-plcg"
server_key: "430bdcdcc76a16964007b8188461a40a"
server_key_iv: "66ff88a1c535b8dc7fbc49e561372061"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:33.688176 12735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:56:33.705947 12750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:33.707623 12735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:56:33.707842 12735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "70afcfffeb2d4eb0b5c460f544136625"
format_stamp: "Formatted at 2026-09-23 20:56:33 on dist-test-slave-plcg"
server_key: "430bdcdcc76a16964007b8188461a40a"
server_key_iv: "66ff88a1c535b8dc7fbc49e561372061"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:33.708078 12735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:33.726724 12735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:33.731276 12735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:33.731626 12735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:33.751807 12735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44171
I20260923 20:56:33.751829 12801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44171 every 8 connection(s)
I20260923 20:56:33.753278 12735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:56:33.757332 12802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:33.760375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 12735
I20260923 20:56:33.760613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:56:33.761561 388 external_mini_cluster.cc:1428] Setting key 6921f6f6ed403cbc6a2d9232ae4b8e20
I20260923 20:56:33.771536 12802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Bootstrap starting.
I20260923 20:56:33.775031 12802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:33.776114 12802 log.cc:824] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:56:33 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:33.779855 12802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: No bootstrap required, opened a new log
I20260923 20:56:33.787945 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:33.764084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.0.97.62:44171)
0923 20:56:33.765080 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:33.765095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:33.765144 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:56:33.767479 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:33.767511 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:33.768004 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:33.768515 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:33.770105 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:33.771497 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:33.772873 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:33.773638 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:33.773888 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:33.777906 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:33.777919 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:33.777951 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:33.778001 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:33.782327 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:33.783593 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:33.783604 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:33.783618 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:33.783872 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:33.784432 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:33.784439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:33.784445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:33.785306 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:33.785618 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:33.786354 (+ 736us) server_negotiation.cc:299] Negotiation successful
0923 20:56:33.786811 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":427,"threads_started":1}
I20260923 20:56:33.790370 12802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:33.790817 12802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:33.790928 12802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70afcfffeb2d4eb0b5c460f544136625, State: Initialized, Role: FOLLOWER
I20260923 20:56:33.791519 12802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:33.791790 12802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:56:33.791934 12802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:56:33.792101 12802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:33.794417 12802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:33.794996 12802 leader_election.cc:302] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70afcfffeb2d4eb0b5c460f544136625; no voters:
I20260923 20:56:33.795686 12802 leader_election.cc:288] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:56:33.795897 12807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:56:33.797421 12807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 LEADER]: Becoming Leader. State: Replica: 70afcfffeb2d4eb0b5c460f544136625, State: Running, Role: LEADER
I20260923 20:56:33.798058 12807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:33.798375 12802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:33.801198 12809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70afcfffeb2d4eb0b5c460f544136625. Latest consensus state: current_term: 1 leader_uuid: "70afcfffeb2d4eb0b5c460f544136625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } } }
I20260923 20:56:33.801496 12808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70afcfffeb2d4eb0b5c460f544136625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } } }
I20260923 20:56:33.801736 12809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:33.802156 12808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:33.802842 12814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:56:33.809096 12814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:56:33.820168 12802 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:56:33.821597 12814 catalog_manager.cc:1347] Generated new cluster ID: ce8dfe6b75cf40f1958b6962481b3684
I20260923 20:56:33.821723 12814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:56:33.848635 12814 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:56:33.850870 12814 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:56:33.861729 12814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Generated new TSK 0
I20260923 20:56:33.863006 12814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:56:34.670811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44171
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:56:34 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790196994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:56:35.156844 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:35.157203 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:35.157393 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:35.167256 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:35.167472 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:35.167749 12847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:56:35.179127 12847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:60455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:35.180591 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:35.182149 12847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:35.198130 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:35.198103 12858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:35.199637 12857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:35.199759 12847 server_base.cc:1034] running on GCE node
I20260923 20:56:35.200604 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:35.214231 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:35.215654 12847 hybrid_clock.cc:648] HybridClock initialized: now 1790196995215578 us; error 79 us; skew 500 ppm
I20260923 20:56:35.216112 12847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:35 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:35.220775 12847 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:56:35.224350 12847 webserver.cc:460] Webserver started at http://127.0.97.1:33269/ using document root <none> and password file <none>
I20260923 20:56:35.225343 12847 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:35.225500 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:35.225900 12847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:56:35 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:35.255899 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "b44c59d246c749dd9c3bfa3ac240f765"
format_stamp: "Formatted at 2026-09-23 20:56:35 on dist-test-slave-plcg"
server_key: "39c5ac942c0f1cebb2fa9b1e01ab6ccc"
server_key_iv: "e904d1fd38a0e2281c3733dab9be7c6e"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:35.257318 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b44c59d246c749dd9c3bfa3ac240f765"
format_stamp: "Formatted at 2026-09-23 20:56:35 on dist-test-slave-plcg"
server_key: "39c5ac942c0f1cebb2fa9b1e01ab6ccc"
server_key_iv: "e904d1fd38a0e2281c3733dab9be7c6e"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:35.267365 12847 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260923 20:56:35.306794 12867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:35.309149 12847 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.008s sys 0.008s
I20260923 20:56:35.309409 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "b44c59d246c749dd9c3bfa3ac240f765"
format_stamp: "Formatted at 2026-09-23 20:56:35 on dist-test-slave-plcg"
server_key: "39c5ac942c0f1cebb2fa9b1e01ab6ccc"
server_key_iv: "e904d1fd38a0e2281c3733dab9be7c6e"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:35.309697 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:35.374302 12847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:35.380206 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:35.380648 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:35.383000 12847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:35.385695 12847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:35.385887 12847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:56:35.386042 12847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:35.386119 12847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:56:35 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:35.496901 12847 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39875
I20260923 20:56:35.496907 12979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39875 every 8 connection(s)
I20260923 20:56:35.498445 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:56:35.499740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 12847
I20260923 20:56:35.499938 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:56:35.500674 388 external_mini_cluster.cc:1428] Setting key 13ef86be062536c198d0b1342b8146e6
Sep 23 20:56:35 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:56:35.534046 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:35.503121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41329 (local address 127.0.97.62:44171)
0923 20:56:35.503628 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:35.503637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:35.507946 (+ 4309us) server_negotiation.cc:407] Connection header received
0923 20:56:35.509915 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:35.509920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:35.510051 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:35.510185 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:35.515117 (+ 4932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:35.516475 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:35.519408 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:35.520204 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:35.520329 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:35.525545 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:35.525555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:35.525561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:35.525627 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:35.529884 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:35.531157 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:35.531164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:35.531168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:35.531352 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:35.532012 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:35.532017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:35.532026 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:35.532310 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:35.532485 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:35.533304 (+ 819us) server_negotiation.cc:299] Negotiation successful
0923 20:56:35.533634 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":243,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 20:56:35.535303 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:35.502460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44171 (local address 127.0.97.1:41329)
0923 20:56:35.507737 (+ 5277us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:35.507798 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:35.509558 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:35.512094 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:35.512114 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:35.513866 (+ 1752us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:35.514840 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:35.514866 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:35.516734 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:35.516743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:35.519183 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:35.519206 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:35.520724 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:35.520734 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:35.521102 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:35.522016 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:35.522069 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:35.525212 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:35.530173 (+ 4961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:35.530183 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:35.530205 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:35.530960 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:35.531542 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:35.531546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:35.531551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:35.531869 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:35.532455 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:35.532464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:35.532900 (+ 436us) client_negotiation.cc:769] Sending connection context
0923 20:56:35.533195 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 20:56:35.533750 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4333,"thread_start_us":4076,"threads_started":1}
I20260923 20:56:35.539821 12980 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44171
I20260923 20:56:35.540192 12980 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:35.541115 12980 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
I20260923 20:56:35.544050 12767 ts_manager.cc:194] Registered new tserver with Master: b44c59d246c749dd9c3bfa3ac240f765 (127.0.97.1:39875)
I20260923 20:56:35.546479 12767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:56:35.567588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44171
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:56:35 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 20:56:36.071409 12986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:36.071940 12986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:36.072144 12986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:36.083244 12986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:36.083436 12986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:36.083717 12986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:56:36.099471 12986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:60455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:36.101342 12986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:36.103114 12986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:36.121151 12993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:36.121863 12994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:36.129189 12996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:36.130177 12986 server_base.cc:1034] running on GCE node
I20260923 20:56:36.131076 12986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:36.132441 12986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:36.133770 12986 hybrid_clock.cc:648] HybridClock initialized: now 1790196996133697 us; error 61 us; skew 500 ppm
I20260923 20:56:36.134253 12986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:36 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:36.138846 12986 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:56:36.141000 12986 webserver.cc:460] Webserver started at http://127.0.97.2:33401/ using document root <none> and password file <none>
I20260923 20:56:36.142024 12986 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:36.142181 12986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:36.142525 12986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:56:36 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:36.162518 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "811ccfa9daae46418bec361073aab680"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "300c68dbf9afa64c03ca4d97a66b4216"
server_key_iv: "f038f57d896df742913fb8c22b192829"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:36.163831 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "811ccfa9daae46418bec361073aab680"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "300c68dbf9afa64c03ca4d97a66b4216"
server_key_iv: "f038f57d896df742913fb8c22b192829"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:36.180725 12986 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.011s sys 0.004s
I20260923 20:56:36.201985 13010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:36.204025 12986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 20:56:36.204339 12986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "811ccfa9daae46418bec361073aab680"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "300c68dbf9afa64c03ca4d97a66b4216"
server_key_iv: "f038f57d896df742913fb8c22b192829"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:36.204681 12986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:36.232561 12986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:36.237910 12986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:36.238255 12986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:36.239444 12986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:36.241283 12986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:36.241385 12986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:36.241514 12986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:36.241578 12986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:36.275455 12986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35927
I20260923 20:56:36.275468 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35927 every 8 connection(s)
I20260923 20:56:36.277185 12986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:56:36.280750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 12986
I20260923 20:56:36.280930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:56:36.281510 388 external_mini_cluster.cc:1428] Setting key 1a2642f1d3858c6629e067bd8c41683c
Sep 23 20:56:36 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:36.304039 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:36.280008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38405 (local address 127.0.97.62:44171)
0923 20:56:36.280526 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:36.280535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:36.281646 (+ 1111us) server_negotiation.cc:407] Connection header received
0923 20:56:36.283981 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:36.283987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:36.284094 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:36.284213 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:36.286404 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.287562 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.290368 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.291033 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.291202 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:36.295380 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:36.295389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:36.295397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:36.295449 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:36.299635 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:36.300817 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:36.300825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:36.300832 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:36.301005 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:36.301602 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:36.301610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:36.301617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:36.302075 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:36.302243 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:36.303224 (+ 981us) server_negotiation.cc:299] Negotiation successful
0923 20:56:36.303594 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":296,"threads_started":1}
I20260923 20:56:36.305222 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:36.280568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44171 (local address 127.0.97.2:38405)
0923 20:56:36.281466 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:36.281515 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:36.283650 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:36.284530 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:36.284543 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:36.285198 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:36.286169 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:36.286202 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.287760 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.287769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:36.290151 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:36.290174 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.291334 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.291345 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:36.291654 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:36.292396 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:36.292421 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:36.295036 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:36.299865 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:36.299876 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:36.299898 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:36.300633 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:36.301159 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:36.301165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:36.301169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:36.301445 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:36.302227 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:36.302237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:36.302726 (+ 489us) client_negotiation.cc:769] Sending connection context
0923 20:56:36.303073 (+ 347us) client_negotiation.cc:240] Negotiation successful
0923 20:56:36.303720 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":364,"threads_started":1}
I20260923 20:56:36.306941 13123 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44171
I20260923 20:56:36.307329 13123 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:56:36.308315 13123 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
I20260923 20:56:36.310118 12767 ts_manager.cc:194] Registered new tserver with Master: 811ccfa9daae46418bec361073aab680 (127.0.97.2:35927)
I20260923 20:56:36.311336 12767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:56:36.342077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44171
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:56:36.550808 12980 heartbeater.cc:503] Master 127.0.97.62:44171 was elected leader, sending a full tablet report...
W20260923 20:56:36.688894 13129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:36.689253 13129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:36.689409 13129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:36.697192 13129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:36.697371 13129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:36.697537 13129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:56:36.708179 13129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:60455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:36.709619 13129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:36.711149 13129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:36.725349 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:36.725373 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:36.726141 13137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:36.726485 13129 server_base.cc:1034] running on GCE node
I20260923 20:56:36.727108 13129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:36.728111 13129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:36.729324 13129 hybrid_clock.cc:648] HybridClock initialized: now 1790196996729282 us; error 27 us; skew 500 ppm
I20260923 20:56:36.729697 13129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:36 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:36.732968 13129 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:56:36.734732 13129 webserver.cc:460] Webserver started at http://127.0.97.3:33205/ using document root <none> and password file <none>
I20260923 20:56:36.735481 13129 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:36.735642 13129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:36.735968 13129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:56:36 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:36.754057 13129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "7bb32f5af8904bd89863b523c717fdb6"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "f7e276e83e2012a1a2c74ab6be903afb"
server_key_iv: "591268e554f73459032094a25c52ed07"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:36.755308 13129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7bb32f5af8904bd89863b523c717fdb6"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "f7e276e83e2012a1a2c74ab6be903afb"
server_key_iv: "591268e554f73459032094a25c52ed07"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:36.762889 13129 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260923 20:56:36.782580 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:36.784220 13129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:56:36.784430 13129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "7bb32f5af8904bd89863b523c717fdb6"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "f7e276e83e2012a1a2c74ab6be903afb"
server_key_iv: "591268e554f73459032094a25c52ed07"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:36.784667 13129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:36.804829 13129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:36.809322 13129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:36.809648 13129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:36.810812 13129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:36.812503 13129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:36.812605 13129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:36.812750 13129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:36.812819 13129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:36.846608 13129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36117
I20260923 20:56:36.846645 13256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36117 every 8 connection(s)
I20260923 20:56:36.847971 13129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:56:36.856549 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 13129
I20260923 20:56:36.856747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:56:36.857247 388 external_mini_cluster.cc:1428] Setting key ddc85cc2140a388b88ed609c94ba10d1
Sep 23 20:56:36 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:36.869525 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:36.850940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50879 (local address 127.0.97.62:44171)
0923 20:56:36.851537 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:36.851542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:36.852370 (+ 828us) server_negotiation.cc:407] Connection header received
0923 20:56:36.854288 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:36.854294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:36.854402 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:36.854528 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:36.856074 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.856947 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.859057 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.859676 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.859813 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:36.863399 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:36.863407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:36.863424 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:36.863472 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:36.866237 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:36.867094 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:36.867099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:36.867101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:36.867235 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:36.867695 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:36.867699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:36.867701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:36.867996 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:36.868156 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:36.868881 (+ 725us) server_negotiation.cc:299] Negotiation successful
0923 20:56:36.869152 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260923 20:56:36.870590 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:36.851421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44171 (local address 127.0.97.3:50879)
0923 20:56:36.852192 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:36.852233 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:36.853982 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:36.854791 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:36.854801 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:36.855264 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:36.855881 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:36.855898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.857120 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.857123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:36.858889 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:36.858905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.859940 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.859947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:36.860198 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:36.860888 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:36.860912 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:36.863105 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:36.866423 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:36.866434 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:36.866455 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:36.866937 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:36.867353 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:36.867358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:36.867362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:36.867561 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:36.868129 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:36.868138 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:36.868496 (+ 358us) client_negotiation.cc:769] Sending connection context
0923 20:56:36.868783 (+ 287us) client_negotiation.cc:240] Negotiation successful
0923 20:56:36.869363 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":327,"threads_started":1}
I20260923 20:56:36.872210 13257 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44171
I20260923 20:56:36.872539 13257 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:36.873256 13257 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
I20260923 20:56:36.874838 12766 ts_manager.cc:194] Registered new tserver with Master: 7bb32f5af8904bd89863b523c717fdb6 (127.0.97.3:36117)
I20260923 20:56:36.875794 12766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50879
I20260923 20:56:36.885790 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:56:36.887476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 12847
Sep 23 20:56:36 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:56:36.904773 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:36.888533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.0.97.1:39875)
0923 20:56:36.889230 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:36.889237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:36.889266 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:56:36.889350 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:36.889356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:36.889585 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:36.889714 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:36.891234 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.892262 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.893725 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:36.894415 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:36.894554 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:36.899113 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:36.899130 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:36.899146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:36.899199 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:36.902265 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:36.902991 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:36.902999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:36.903001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:36.903151 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:36.903465 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:36.903468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:36.903470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:36.903802 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:36.904029 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:36.904270 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 20:56:36.904475 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":376,"threads_started":1}
I20260923 20:56:36.906141 12975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:36.991058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 12847
I20260923 20:56:37.007462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 12986
Sep 23 20:56:37 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:56:37.024837 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:37.008410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.0.97.2:35927)
0923 20:56:37.008986 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:37.008994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:37.009021 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:56:37.009083 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:37.009091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:37.009313 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:37.009447 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:37.010824 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:37.012012 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:37.013424 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:37.014119 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:37.014235 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:37.019013 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:37.019030 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:37.019047 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:37.019088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:37.021985 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:37.022867 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:37.022875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:37.022877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:37.023021 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:37.023388 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:37.023391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:37.023393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:37.023734 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:37.023960 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:37.024180 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 20:56:37.024390 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":302,"threads_started":1}
I20260923 20:56:37.026146 13118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:37.114163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 12986
I20260923 20:56:37.138998 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 13129
Sep 23 20:56:37 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:56:37.160168 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:37.139938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.0.97.3:36117)
0923 20:56:37.140609 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:37.140622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:37.140665 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:56:37.140754 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:37.140764 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:37.141119 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:37.141320 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:37.142937 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:37.144567 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:37.146532 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:37.147417 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:37.147555 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:37.153649 (+ 6094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:37.153674 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:37.153701 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:37.153771 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:37.157346 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:37.158250 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:37.158257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:37.158259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:37.158399 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:37.158830 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:37.158833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:37.158835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:37.159177 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:37.159405 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:37.159679 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 20:56:37.159884 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":290,"threads_started":1}
I20260923 20:56:37.161427 13252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:37.246378 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 13129
I20260923 20:56:37.262054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 12735
I20260923 20:56:37.263343 12797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:37.370609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 12735
I20260923 20:56:37.397030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44171
--webserver_interface=127.0.97.62
--webserver_port=45755
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44171
--ranger_config_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:37.730978 13269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:37.731282 13269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:37.731439 13269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:37.738909 13269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:37.739034 13269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:37.739113 13269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:37.739198 13269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:37.739290 13269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:37.749930 13269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44171
--ranger_config_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:37.751298 13269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:37.752667 13269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:37.764849 13277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:37.764964 13274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:37.764984 13275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:37.765916 13269 server_base.cc:1034] running on GCE node
I20260923 20:56:37.766712 13269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:37.768251 13269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:37.769480 13269 hybrid_clock.cc:648] HybridClock initialized: now 1790196997769445 us; error 50 us; skew 500 ppm
I20260923 20:56:37.769901 13269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:37 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:37.773169 13269 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:56:37.774842 13269 webserver.cc:460] Webserver started at http://127.0.97.62:45755/ using document root <none> and password file <none>
I20260923 20:56:37.775535 13269 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:37.775691 13269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:56:37 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:37.834666 13269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:56:37.837993 13285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:37.839627 13269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:56:37.839846 13269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "70afcfffeb2d4eb0b5c460f544136625"
format_stamp: "Formatted at 2026-09-23 20:56:33 on dist-test-slave-plcg"
server_key: "430bdcdcc76a16964007b8188461a40a"
server_key_iv: "66ff88a1c535b8dc7fbc49e561372061"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:37.840894 13269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:37.875654 13269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:37.879954 13269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:37.880290 13269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:37.902987 13269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44171
I20260923 20:56:37.902997 13336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44171 every 8 connection(s)
I20260923 20:56:37.904949 13269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:56:37.911677 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 13269
I20260923 20:56:37.912169 13337 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:56:37.912602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39875
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44171
--webserver_port=33269
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:56:37.915760 13337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Bootstrap starting.
I20260923 20:56:37.934882 13337 log.cc:824] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:37.944022 13337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:56:37.944619 13337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Bootstrap complete.
I20260923 20:56:37.954061 13337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:37.955008 13337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70afcfffeb2d4eb0b5c460f544136625, State: Initialized, Role: FOLLOWER
I20260923 20:56:37.955644 13337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:37.955901 13337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:56:37.956036 13337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:56:37.956221 13337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:56:37.960400 13337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:37.960956 13337 leader_election.cc:302] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70afcfffeb2d4eb0b5c460f544136625; no voters:
I20260923 20:56:37.961632 13337 leader_election.cc:288] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:56:37.961772 13341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:56:37.963057 13341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [term 2 LEADER]: Becoming Leader. State: Replica: 70afcfffeb2d4eb0b5c460f544136625, State: Running, Role: LEADER
I20260923 20:56:37.963708 13341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } }
I20260923 20:56:37.964104 13337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:37.966317 13343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70afcfffeb2d4eb0b5c460f544136625. Latest consensus state: current_term: 2 leader_uuid: "70afcfffeb2d4eb0b5c460f544136625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } } }
I20260923 20:56:37.967095 13343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:37.967093 13342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "70afcfffeb2d4eb0b5c460f544136625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70afcfffeb2d4eb0b5c460f544136625" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44171 } } }
I20260923 20:56:37.968391 13342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:37.978891 13347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:56:37.979830 13337 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:56:37.985047 13347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:56:37.986683 13347 catalog_manager.cc:1259] Loaded cluster ID: ce8dfe6b75cf40f1958b6962481b3684
I20260923 20:56:37.986882 13347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:56:37.991201 13347 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:56:37.992991 13347 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Loaded TSK: 0
I20260923 20:56:38.000564 13347 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:56:38.385211 13339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:38.385771 13339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:38.386018 13339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:38.396136 13339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:38.396407 13339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:38.396654 13339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:56:38.410328 13339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33269
--tserver_master_addrs=127.0.97.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:38.411986 13339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:38.413609 13339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:38.439906 13374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:38.440872 13375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:38.455250 13339 server_base.cc:1034] running on GCE node
W20260923 20:56:38.455976 13377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:38.456916 13339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:38.458595 13339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:38.459911 13339 hybrid_clock.cc:648] HybridClock initialized: now 1790196998459836 us; error 61 us; skew 500 ppm
I20260923 20:56:38.460386 13339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:38 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:38.474939 13339 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:56:38.487412 13339 webserver.cc:460] Webserver started at http://127.0.97.1:33269/ using document root <none> and password file <none>
I20260923 20:56:38.488325 13339 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:38.488519 13339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:56:38 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:38.519284 13339 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.005s sys 0.001s
I20260923 20:56:38.522842 13385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:38.524875 13339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:56:38.525080 13339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "b44c59d246c749dd9c3bfa3ac240f765"
format_stamp: "Formatted at 2026-09-23 20:56:35 on dist-test-slave-plcg"
server_key: "39c5ac942c0f1cebb2fa9b1e01ab6ccc"
server_key_iv: "e904d1fd38a0e2281c3733dab9be7c6e"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:38.526299 13339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:38.571717 13339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:38.577534 13339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:38.577909 13339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:38.581120 13339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:38.583056 13339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:38.583156 13339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:38.583287 13339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:38.583380 13339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:38.642064 13339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39875
I20260923 20:56:38.642169 13498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39875 every 8 connection(s)
I20260923 20:56:38.643563 13339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:56:38.648324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 13339
I20260923 20:56:38.649163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35927
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44171
--webserver_port=33401
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:56:38 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:38.680080 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:38.647383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44171 (local address 127.0.97.1:50041)
0923 20:56:38.648259 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:38.648313 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:38.651158 (+ 2845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:38.657377 (+ 6219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:38.657391 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:38.658120 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:38.658985 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:38.659017 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:38.661667 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:38.661675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:38.663865 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:38.663887 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:38.665618 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:38.665628 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:38.665966 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:38.666832 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:38.666865 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:38.669771 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:38.674606 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:38.674615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:38.674632 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:38.675280 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:38.675880 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:38.675885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:38.675889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:38.676146 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:38.677198 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:38.677207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:38.677627 (+ 420us) client_negotiation.cc:769] Sending connection context
0923 20:56:38.677979 (+ 352us) client_negotiation.cc:240] Negotiation successful
0923 20:56:38.678599 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":6429,"threads_started":1}
I20260923 20:56:38.683877 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:38.647678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50041 (local address 127.0.97.62:44171)
0923 20:56:38.655381 (+ 7703us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:38.655403 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:38.655476 (+ 73us) server_negotiation.cc:407] Connection header received
0923 20:56:38.655975 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:38.656000 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:38.656629 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:38.657139 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:38.659323 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:38.661399 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:38.664168 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:38.665295 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:38.666508 (+ 1213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:38.670190 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:38.670201 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:38.670224 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:38.670293 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:38.674384 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:38.675496 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:38.675506 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:38.675514 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:38.675731 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:38.676323 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:38.676328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:38.676330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:38.677017 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:38.681614 (+ 4597us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:38.681954 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 20:56:38.682446 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7378,"thread_start_us":8297,"threads_started":1}
I20260923 20:56:38.691002 13499 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44171
I20260923 20:56:38.691435 13499 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:38.692466 13499 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
W20260923 20:56:38.696419 13499 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44171 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:56:38.696811 13499 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:38.697163 13499 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
I20260923 20:56:38.698921 13499 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:38.699343 13499 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
W20260923 20:56:38.700682 13499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:56:39.063397 13503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:39.063760 13503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:39.063934 13503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:39.074574 13503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:39.074744 13503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:39.074924 13503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:56:39.085793 13503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33401
--tserver_master_addrs=127.0.97.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:39.087366 13503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:39.088721 13503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:39.103722 13518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:39.107533 13520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:39.107915 13517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:39.109347 13503 server_base.cc:1034] running on GCE node
I20260923 20:56:39.110180 13503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:39.111286 13503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:39.112514 13503 hybrid_clock.cc:648] HybridClock initialized: now 1790196999112448 us; error 67 us; skew 500 ppm
I20260923 20:56:39.112900 13503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:39 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:39.116441 13503 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:56:39.118477 13503 webserver.cc:460] Webserver started at http://127.0.97.2:33401/ using document root <none> and password file <none>
I20260923 20:56:39.119223 13503 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:39.119369 13503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:56:39 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:39.166994 13503 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.003s sys 0.004s
I20260923 20:56:39.170626 13530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:39.172490 13503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:56:39.172693 13503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "811ccfa9daae46418bec361073aab680"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "300c68dbf9afa64c03ca4d97a66b4216"
server_key_iv: "f038f57d896df742913fb8c22b192829"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:39.174199 13503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:39.238781 13503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:39.244383 13503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:39.244773 13503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:39.252293 13503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:39.260665 13503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:39.260818 13503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:39.260946 13503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:39.261023 13503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:56:39 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790196999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:39.562881 13503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35927
I20260923 20:56:39.562954 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35927 every 8 connection(s)
I20260923 20:56:39.564772 13503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:56:39.569648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 13503
I20260923 20:56:39.570483 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36117
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44171
--webserver_port=33205
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:56:39 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:39.657455 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:39.586855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33085 (local address 127.0.97.62:44171)
0923 20:56:39.612159 (+ 25304us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:39.612167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:39.613886 (+ 1719us) server_negotiation.cc:407] Connection header received
0923 20:56:39.617242 (+ 3356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:39.617249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:39.617354 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:39.617442 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:39.619876 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:39.620892 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:39.623369 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:39.624008 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:39.624117 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:39.646238 (+ 22121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:39.646246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:39.646251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:39.646300 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:39.649937 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:39.654362 (+ 4425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:39.654370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:39.654374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:39.654592 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:39.655247 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:39.655252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:39.655256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:39.655644 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:39.655791 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:39.656799 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0923 20:56:39.657077 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25121,"thread_start_us":25327,"threads_started":1}
I20260923 20:56:39.659827 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:39.586125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44171 (local address 127.0.97.2:33085)
0923 20:56:39.613039 (+ 26914us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:39.613104 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:39.615821 (+ 2717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:39.618098 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:39.618111 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:39.618786 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:39.619627 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:39.619669 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:39.621158 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:39.621163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:39.623157 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:39.623176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:39.624512 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:39.624519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:39.624824 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:39.626244 (+ 1420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:39.626280 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:39.645750 (+ 19470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:39.650185 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:39.650195 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:39.650212 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:39.650857 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:39.654794 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:39.654800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:39.654803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:39.655091 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:39.655893 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:39.655901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:39.656325 (+ 424us) client_negotiation.cc:769] Sending connection context
0923 20:56:39.657807 (+ 1482us) client_negotiation.cc:240] Negotiation successful
0923 20:56:39.658414 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26526,"thread_start_us":29976,"threads_started":1}
I20260923 20:56:39.661664 13644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44171
I20260923 20:56:39.662055 13644 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:39.662963 13644 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
I20260923 20:56:39.665189 13302 ts_manager.cc:194] Registered new tserver with Master: 811ccfa9daae46418bec361073aab680 (127.0.97.2:35927)
I20260923 20:56:39.667485 13302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33085
I20260923 20:56:39.700834 13499 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:39.701292 13499 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
I20260923 20:56:39.702823 13302 ts_manager.cc:194] Registered new tserver with Master: b44c59d246c749dd9c3bfa3ac240f765 (127.0.97.1:39875)
I20260923 20:56:39.704196 13302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50041
Sep 23 20:56:39 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 20:56:40.096606 13645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:40.097025 13645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:40.097241 13645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:40.108811 13645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:40.108975 13645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:40.109217 13645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:56:40.125115 13645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33205
--tserver_master_addrs=127.0.97.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:40.127110 13645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:40.128815 13645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:40.146790 13654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:40.146867 13653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:40.155768 13645 server_base.cc:1034] running on GCE node
W20260923 20:56:40.156102 13656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:40.156965 13645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:40.158434 13645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:40.161092 13645 hybrid_clock.cc:648] HybridClock initialized: now 1790197000160976 us; error 90 us; skew 500 ppm
I20260923 20:56:40.161579 13645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:40 dist-test-slave-plcg krb5kdc[12291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:40.166332 13645 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:56:40.171849 13645 webserver.cc:460] Webserver started at http://127.0.97.3:33205/ using document root <none> and password file <none>
I20260923 20:56:40.172986 13645 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:40.173192 13645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:56:40 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:40.218613 13645 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.006s sys 0.005s
I20260923 20:56:40.223069 13665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:40.225140 13645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:56:40.225364 13645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "7bb32f5af8904bd89863b523c717fdb6"
format_stamp: "Formatted at 2026-09-23 20:56:36 on dist-test-slave-plcg"
server_key: "f7e276e83e2012a1a2c74ab6be903afb"
server_key_iv: "591268e554f73459032094a25c52ed07"
server_key_version: "kuduclusterkey@0"
I20260923 20:56:40.226890 13645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 23 20:56:40 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790196999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 20:56:40.294782 13645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:40.300024 13645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:40.300400 13645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:40.331003 13645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:40.346652 13645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:40.346849 13645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 20:56:40.347000 13645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:40.347111 13645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:40.494920 13645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36117
I20260923 20:56:40.494951 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36117 every 8 connection(s)
I20260923 20:56:40.496892 13645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:56:40.499408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 13645
Sep 23 20:56:40 dist-test-slave-plcg krb5kdc[12291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:40.533399 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.500320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.0.97.62:44171)
0923 20:56:40.517526 (+ 17206us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.517533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.517553 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:56:40.517605 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.517610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.517704 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.517810 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:40.519944 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.520979 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.523043 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.523728 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.523863 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.527047 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:40.527056 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:40.527059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:40.527110 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:40.530125 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.531047 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.531051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.531055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.531196 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.531707 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.531713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.531716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.532061 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:40.532336 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.532716 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.533012 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17068,"thread_start_us":13955,"threads_started":1}
I20260923 20:56:40.534317 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.500260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33653 (local address 127.0.97.62:44171)
0923 20:56:40.514491 (+ 14231us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.514497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.514519 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:56:40.514587 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.514592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.514688 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.514798 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:40.517397 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.518223 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.520062 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.520864 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.521009 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.525410 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:40.525417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:40.525422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:40.525478 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:40.529613 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.530738 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.530743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.530746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.530898 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.531459 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.531467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.531472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.531854 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:40.532006 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.533681 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.533922 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14096,"thread_start_us":16947,"threads_started":1}
I20260923 20:56:40.536109 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.500492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44171 (local address 127.0.97.3:33653)
0923 20:56:40.509378 (+ 8886us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:40.509442 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:40.512941 (+ 3499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:40.515308 (+ 2367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:40.515323 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:40.516019 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:40.516908 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.516937 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.518403 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.518409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.519876 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.519896 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.521205 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.521213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.521543 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.522434 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:40.522467 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:40.525113 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:40.529870 (+ 4757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:40.529881 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:40.529903 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:40.530596 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:40.531053 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:40.531059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:40.531063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:40.531322 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:40.531996 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:40.532005 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:40.532622 (+ 617us) client_negotiation.cc:769] Sending connection context
0923 20:56:40.533008 (+ 386us) client_negotiation.cc:240] Negotiation successful
0923 20:56:40.534617 (+ 1609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8489,"thread_start_us":9198,"threads_started":1}
I20260923 20:56:40.537670 13785 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44171
I20260923 20:56:40.538015 13785 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:40.538784 13785 heartbeater.cc:511] Master 127.0.97.62:44171 requested a full tablet report, sending...
I20260923 20:56:40.540504 13302 ts_manager.cc:194] Registered new tserver with Master: 7bb32f5af8904bd89863b523c717fdb6 (127.0.97.3:36117)
I20260923 20:56:40.541354 13302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33653
I20260923 20:56:40.548815 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:56:40.565670 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.552115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.97.62:44171)
0923 20:56:40.552360 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.552368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.552669 (+ 301us) server_negotiation.cc:407] Connection header received
0923 20:56:40.552868 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.552873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.552950 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.553054 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:40.554693 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.555417 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.556784 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.557209 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.557342 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.560014 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:40.560025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:40.560029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:40.560074 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:40.562805 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.563555 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.563561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.563565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.563708 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.564110 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.564117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.564120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.564404 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:40.564684 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.565015 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.565311 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:56:40.570799 13302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:56:40.579094 13302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:56:40.605701 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.595244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.0.97.2:35927)
0923 20:56:40.596558 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.596567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.597362 (+ 795us) server_negotiation.cc:407] Connection header received
0923 20:56:40.597511 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.597520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.597887 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.598110 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:40.599137 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.600598 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.603441 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.604476 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.604614 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.604870 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.605188 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.605365 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":1162,"threads_started":1}
I20260923 20:56:40.605749 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.594849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35927 (local address 127.0.0.1:33168)
0923 20:56:40.597067 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:40.597089 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:40.597225 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:40.598409 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:40.598416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:40.598447 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:40.598904 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.598918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.600747 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.600753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.603277 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.603292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.604706 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.604713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.604913 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.604966 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 20:56:40.605115 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 20:56:40.605322 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2054,"mutex_wait_us":2,"thread_start_us":1904,"threads_started":1}
I20260923 20:56:40.607427 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.594046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36117 (local address 127.0.0.1:58272)
0923 20:56:40.595146 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:40.595196 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:40.595403 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:40.598955 (+ 3552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:40.598962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:40.598989 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:40.599413 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.599424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.601267 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.601274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.604292 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.604308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.606549 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.606556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.606765 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.606800 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:56:40.606869 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 20:56:40.607000 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":130,"thread_start_us":610,"threads_started":1}
I20260923 20:56:40.608201 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.595746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55212)
0923 20:56:40.598056 (+ 2310us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:40.598077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:40.598287 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:40.599179 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:40.599183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:40.599202 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:40.599623 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.599644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.601302 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.601309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.604293 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.604308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.606288 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.606294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.606475 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.606526 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:56:40.607697 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0923 20:56:40.607850 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2176,"mutex_wait_us":98,"thread_start_us":2006,"threads_started":1}
I20260923 20:56:40.608312 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.596279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.0.97.1:39875)
0923 20:56:40.597543 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.597556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.598181 (+ 625us) server_negotiation.cc:407] Connection header received
0923 20:56:40.598480 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.598489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.598724 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.598891 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:40.599817 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.601116 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.604496 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.606070 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.606228 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.606456 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.607766 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.607963 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":824,"threads_started":1}
I20260923 20:56:40.608805 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.594377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.97.3:36117)
0923 20:56:40.598414 (+ 4037us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.598422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.598452 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:56:40.598536 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.598543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.598723 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.599651 (+ 928us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:40.599925 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.601105 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.604476 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.606279 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.607836 (+ 1557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.608075 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.608143 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.608305 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3840,"thread_start_us":4705,"threads_started":1}
I20260923 20:56:40.610924 13578 tablet_service.cc:1467] Processing CreateTablet for tablet bf959016138141c1867bd3ccf83af35b (DEFAULT_TABLE table=test-table [id=0beb639ce0964bc98b39b3b1e288ac5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:40.611871 13713 tablet_service.cc:1467] Processing CreateTablet for tablet bf959016138141c1867bd3ccf83af35b (DEFAULT_TABLE table=test-table [id=0beb639ce0964bc98b39b3b1e288ac5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:40.612082 13434 tablet_service.cc:1467] Processing CreateTablet for tablet bf959016138141c1867bd3ccf83af35b (DEFAULT_TABLE table=test-table [id=0beb639ce0964bc98b39b3b1e288ac5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:40.612689 13578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf959016138141c1867bd3ccf83af35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:40.613216 13713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf959016138141c1867bd3ccf83af35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:40.613404 13434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf959016138141c1867bd3ccf83af35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:40.626691 13806 tablet_bootstrap.cc:492] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Bootstrap starting.
I20260923 20:56:40.628768 13807 tablet_bootstrap.cc:492] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Bootstrap starting.
I20260923 20:56:40.630242 13806 tablet_bootstrap.cc:654] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:40.631820 13806 log.cc:824] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:40.632959 13807 tablet_bootstrap.cc:654] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:40.633615 13808 tablet_bootstrap.cc:492] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765: Bootstrap starting.
I20260923 20:56:40.635253 13807 log.cc:824] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:40.636092 13806 tablet_bootstrap.cc:492] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: No bootstrap required, opened a new log
I20260923 20:56:40.636485 13806 ts_tablet_manager.cc:1397] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260923 20:56:40.637804 13808 tablet_bootstrap.cc:654] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:40.639032 13807 tablet_bootstrap.cc:492] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: No bootstrap required, opened a new log
I20260923 20:56:40.639171 13808 log.cc:824] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:40.639348 13807 ts_tablet_manager.cc:1397] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 20:56:40.642699 13808 tablet_bootstrap.cc:492] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765: No bootstrap required, opened a new log
I20260923 20:56:40.643085 13808 ts_tablet_manager.cc:1397] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260923 20:56:40.646389 13806 raft_consensus.cc:348] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.647020 13806 raft_consensus.cc:374] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:40.647168 13806 raft_consensus.cc:729] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb32f5af8904bd89863b523c717fdb6, State: Initialized, Role: FOLLOWER
I20260923 20:56:40.647993 13806 consensus_queue.cc:260] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.652640 13807 raft_consensus.cc:348] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.653216 13807 raft_consensus.cc:374] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:40.653366 13807 raft_consensus.cc:729] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811ccfa9daae46418bec361073aab680, State: Initialized, Role: FOLLOWER
I20260923 20:56:40.654238 13807 consensus_queue.cc:260] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.654678 13808 raft_consensus.cc:348] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.655445 13785 heartbeater.cc:503] Master 127.0.97.62:44171 was elected leader, sending a full tablet report...
I20260923 20:56:40.656260 13806 ts_tablet_manager.cc:1428] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
I20260923 20:56:40.657297 13808 raft_consensus.cc:374] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:40.657447 13808 raft_consensus.cc:729] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44c59d246c749dd9c3bfa3ac240f765, State: Initialized, Role: FOLLOWER
I20260923 20:56:40.658322 13808 consensus_queue.cc:260] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.662894 13499 heartbeater.cc:503] Master 127.0.97.62:44171 was elected leader, sending a full tablet report...
I20260923 20:56:40.664405 13808 ts_tablet_manager.cc:1428] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765: Time spent starting tablet: real 0.021s user 0.013s sys 0.004s
I20260923 20:56:40.665498 13644 heartbeater.cc:503] Master 127.0.97.62:44171 was elected leader, sending a full tablet report...
I20260923 20:56:40.666266 13807 ts_tablet_manager.cc:1428] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Time spent starting tablet: real 0.027s user 0.013s sys 0.012s
W20260923 20:56:40.748704 13786 tablet.cc:2369] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:56:40.752583 13813 raft_consensus.cc:482] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:56:40.752909 13813 raft_consensus.cc:504] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.754647 13813 leader_election.cc:288] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bb32f5af8904bd89863b523c717fdb6 (127.0.97.3:36117), b44c59d246c749dd9c3bfa3ac240f765 (127.0.97.1:39875)
I20260923 20:56:40.762024 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.755194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45467 (local address 127.0.97.3:36117)
0923 20:56:40.755426 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.755435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.755828 (+ 393us) server_negotiation.cc:407] Connection header received
0923 20:56:40.756061 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.756068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.756159 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.756269 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:40.756929 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.757911 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.759967 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.760950 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.761054 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.761139 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.761581 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.761721 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:56:40.762152 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.755072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36117 (local address 127.0.97.2:45467)
0923 20:56:40.755708 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:40.755729 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:40.755894 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:40.756311 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:40.756316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:40.756353 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:40.756759 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.756772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.758047 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.758053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.759783 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.759796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.761180 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.761187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.761371 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.761411 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:56:40.761526 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:56:40.761675 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":1,"thread_start_us":307,"threads_started":1}
I20260923 20:56:40.762436 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.755932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49635 (local address 127.0.97.1:39875)
0923 20:56:40.756206 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.756211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.756607 (+ 396us) server_negotiation.cc:407] Connection header received
0923 20:56:40.756722 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.756728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.756805 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.756898 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:40.757505 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.758548 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.760458 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.761436 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.761555 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.761662 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.762044 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.762204 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:56:40.762722 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.755719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.97.2:49635)
0923 20:56:40.756496 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:40.756512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:40.756615 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:40.756925 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:40.756929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:40.756954 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:40.757323 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.757334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.758734 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.758738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.760292 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:40.760308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.761659 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.761665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:40.761870 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.761905 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:56:40.762012 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:56:40.762361 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":452,"threads_started":1}
I20260923 20:56:40.763442 13733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf959016138141c1867bd3ccf83af35b" candidate_uuid: "811ccfa9daae46418bec361073aab680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb32f5af8904bd89863b523c717fdb6" is_pre_election: true
I20260923 20:56:40.763644 13454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf959016138141c1867bd3ccf83af35b" candidate_uuid: "811ccfa9daae46418bec361073aab680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44c59d246c749dd9c3bfa3ac240f765" is_pre_election: true
I20260923 20:56:40.764091 13733 raft_consensus.cc:2454] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811ccfa9daae46418bec361073aab680 in term 0.
I20260923 20:56:40.764221 13454 raft_consensus.cc:2454] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811ccfa9daae46418bec361073aab680 in term 0.
I20260923 20:56:40.764801 13533 leader_election.cc:302] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb32f5af8904bd89863b523c717fdb6, 811ccfa9daae46418bec361073aab680; no voters:
I20260923 20:56:40.765237 13813 raft_consensus.cc:2781] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:56:40.765437 13813 raft_consensus.cc:482] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:56:40.765543 13813 raft_consensus.cc:3037] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:40.767822 13813 raft_consensus.cc:504] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.768375 13813 leader_election.cc:288] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [CANDIDATE]: Term 1 election: Requested vote from peers 7bb32f5af8904bd89863b523c717fdb6 (127.0.97.3:36117), b44c59d246c749dd9c3bfa3ac240f765 (127.0.97.1:39875)
I20260923 20:56:40.768913 13733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf959016138141c1867bd3ccf83af35b" candidate_uuid: "811ccfa9daae46418bec361073aab680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb32f5af8904bd89863b523c717fdb6"
I20260923 20:56:40.769126 13733 raft_consensus.cc:3037] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:40.769301 13454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf959016138141c1867bd3ccf83af35b" candidate_uuid: "811ccfa9daae46418bec361073aab680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44c59d246c749dd9c3bfa3ac240f765"
I20260923 20:56:40.769512 13454 raft_consensus.cc:3037] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:40.771466 13733 raft_consensus.cc:2454] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811ccfa9daae46418bec361073aab680 in term 1.
I20260923 20:56:40.771845 13454 raft_consensus.cc:2454] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811ccfa9daae46418bec361073aab680 in term 1.
I20260923 20:56:40.771944 13533 leader_election.cc:302] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb32f5af8904bd89863b523c717fdb6, 811ccfa9daae46418bec361073aab680; no voters:
I20260923 20:56:40.772298 13813 raft_consensus.cc:2781] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:56:40.772780 13813 raft_consensus.cc:686] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 1 LEADER]: Becoming Leader. State: Replica: 811ccfa9daae46418bec361073aab680, State: Running, Role: LEADER
I20260923 20:56:40.773263 13813 consensus_queue.cc:237] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } }
I20260923 20:56:40.777891 13301 catalog_manager.cc:5495] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 reported cstate change: term changed from 0 to 1, leader changed from <none> to 811ccfa9daae46418bec361073aab680 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "811ccfa9daae46418bec361073aab680" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811ccfa9daae46418bec361073aab680" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35927 } health_report { overall_health: HEALTHY } } }
I20260923 20:56:40.798904 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.793738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.0.97.2:35927)
0923 20:56:40.793959 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.793964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.794004 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:56:40.794217 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.794221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.794309 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.794437 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:56:40.794762 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.795475 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.796840 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.797231 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.797329 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.797795 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:56:40.798402 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:56:40.798526 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.798617 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.798691 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 20:56:40.807459 13733 raft_consensus.cc:1261] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 1 FOLLOWER]: Refusing update from remote peer 811ccfa9daae46418bec361073aab680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:40.807459 13454 raft_consensus.cc:1261] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765 [term 1 FOLLOWER]: Refusing update from remote peer 811ccfa9daae46418bec361073aab680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:40.808518 13817 consensus_queue.cc:1035] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44c59d246c749dd9c3bfa3ac240f765" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:40.808918 13813 consensus_queue.cc:1035] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb32f5af8904bd89863b523c717fdb6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:40.835430 13301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:56:40.835723 13301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:56:40.839738 13301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70afcfffeb2d4eb0b5c460f544136625: Sending DeleteTablet for 3 replicas of tablet bf959016138141c1867bd3ccf83af35b
I20260923 20:56:40.841030 13578 tablet_service.cc:1514] Processing DeleteTablet for tablet bf959016138141c1867bd3ccf83af35b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33168
I20260923 20:56:40.841017 13713 tablet_service.cc:1514] Processing DeleteTablet for tablet bf959016138141c1867bd3ccf83af35b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58272
I20260923 20:56:40.842096 13434 tablet_service.cc:1514] Processing DeleteTablet for tablet bf959016138141c1867bd3ccf83af35b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55212
I20260923 20:56:40.850569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 13339
I20260923 20:56:40.853919 13830 tablet_replica.cc:330] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: stopping tablet replica
I20260923 20:56:40.854449 13830 raft_consensus.cc:2229] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:56:40.855192 13830 raft_consensus.cc:2258] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:56:40.855952 13829 tablet_replica.cc:330] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: stopping tablet replica
I20260923 20:56:40.856433 13829 raft_consensus.cc:2229] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:56:40.856942 13829 raft_consensus.cc:2258] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:56:40.859206 13829 ts_tablet_manager.cc:1905] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:56:40.867173 13829 ts_tablet_manager.cc:1918] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:56:40.867367 13829 log.cc:1196] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/bf959016138141c1867bd3ccf83af35b
I20260923 20:56:40.867928 13829 ts_tablet_manager.cc:1939] T bf959016138141c1867bd3ccf83af35b P 7bb32f5af8904bd89863b523c717fdb6: Deleting consensus metadata
I20260923 20:56:40.870033 13288 catalog_manager.cc:4841] TS 7bb32f5af8904bd89863b523c717fdb6 (127.0.97.3:36117): tablet bf959016138141c1867bd3ccf83af35b (table test-table [id=0beb639ce0964bc98b39b3b1e288ac5f]) successfully deleted
I20260923 20:56:40.871389 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:40.851696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.97.1:39875)
0923 20:56:40.851934 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:40.851943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:40.851970 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:56:40.852054 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:40.852063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:40.852191 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:40.852332 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:40.854724 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.855879 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.858025 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:40.858710 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:40.858852 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:40.862716 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:40.862736 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:40.862756 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:40.862813 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:40.867129 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.868234 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.868243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.868246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.868450 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:40.868913 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:40.868918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:40.868922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:40.869392 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:40.869730 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:40.870924 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0923 20:56:40.871102 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:56:40.872206 13494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:40.874640 13830 ts_tablet_manager.cc:1905] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:56:40.881012 13830 ts_tablet_manager.cc:1918] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:56:40.881230 13830 log.cc:1196] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/bf959016138141c1867bd3ccf83af35b
I20260923 20:56:40.881702 13830 ts_tablet_manager.cc:1939] T bf959016138141c1867bd3ccf83af35b P 811ccfa9daae46418bec361073aab680: Deleting consensus metadata
I20260923 20:56:40.883478 13289 catalog_manager.cc:4841] TS 811ccfa9daae46418bec361073aab680 (127.0.97.2:35927): tablet bf959016138141c1867bd3ccf83af35b (table test-table [id=0beb639ce0964bc98b39b3b1e288ac5f]) successfully deleted
I20260923 20:56:41.002420 13831 tablet_replica.cc:330] T bf959016138141c1867bd3ccf83af35b P b44c59d246c749dd9c3bfa3ac240f765: stopping tablet replica
I20260923 20:56:41.002687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 13339
I20260923 20:56:41.020385 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 13503
W20260923 20:56:41.020490 13288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39875 (error 108)
I20260923 20:56:41.022959 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.022427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55226)
0923 20:56:41.022628 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.022763 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 20:56:41.023298 13288 catalog_manager.cc:4584] TS b44c59d246c749dd9c3bfa3ac240f765 (127.0.97.1:39875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf959016138141c1867bd3ccf83af35b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
I20260923 20:56:41.037111 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.021208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.0.97.2:35927)
0923 20:56:41.021915 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:41.021923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:41.021944 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:56:41.022017 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:41.022022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:41.022118 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:41.022234 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:41.024105 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:41.024975 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:41.026589 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:41.027069 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:41.027218 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:41.030132 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:41.030150 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:41.030168 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:41.030215 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:41.034064 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:41.034880 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:41.034888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:41.034891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:41.035055 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:41.035438 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:41.035443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:41.035451 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:41.035849 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:41.036180 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:41.036492 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 20:56:41.036753 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260923 20:56:41.037851 13639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:41.077065 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.076521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55230)
0923 20:56:41.076728 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.076891 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:56:41.078178 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.077776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55242)
0923 20:56:41.077951 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.078052 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:56:41.138685 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.138153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55250)
0923 20:56:41.138367 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.138490 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:56:41.139695 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.139345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55266)
0923 20:56:41.139497 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.139590 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:56:41.154398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 13503
I20260923 20:56:41.173576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 13645
I20260923 20:56:41.187983 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.174470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.0.97.3:36117)
0923 20:56:41.174675 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:41.174681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:41.174702 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:56:41.174924 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:41.174928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:41.175050 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:41.175136 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:41.176569 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:41.177387 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:41.179118 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:41.179502 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:41.179605 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:41.182405 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:41.182417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:41.182435 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:41.182487 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:41.185199 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:41.186030 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:41.186038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:41.186040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:41.186181 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:41.186581 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:41.186585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:41.186587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:41.186973 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:41.187217 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:41.187523 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 20:56:41.187693 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:56:41.188617 13780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:41.220598 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.220028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55278)
0923 20:56:41.220281 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.220382 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 20:56:41.221892 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.221380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55294)
0923 20:56:41.221601 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.221695 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260923 20:56:41.277380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 13645
I20260923 20:56:41.295296 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 13269
I20260923 20:56:41.296556 13332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:41.410049 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:41.409349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39875 (local address 127.0.0.1:55300)
0923 20:56:41.409588 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:41.409705 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 20:56:41.410269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 13269
I20260923 20:56:41.436046 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:56:41.771811 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:56:41.772035 388 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:56:42.118299 388 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:56:42.118 UTC [12321] LOG: received smart shutdown request
2026-09-23 20:56:42.122 UTC [12321] LOG: background worker "logical replication launcher" (PID 12330) exited with exit code 1
2026-09-23 20:56:42.122 UTC [12325] LOG: shutting down
2026-09-23 20:56:42.319 UTC [12321] LOG: database system is shut down
2026-09-23T20:56:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56439 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:56:42 dist-test-slave-plcg krb5kdc[13845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:56:42 dist-test-slave-plcg krb5kdc[13845](info): set up 2 sockets
Sep 23 20:56:42 dist-test-slave-plcg krb5kdc[13845](info): commencing operation
krb5kdc: starting...
W20260923 20:56:43.558255 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:56:43 dist-test-slave-plcg krb5kdc[13845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:56:43.797914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35439
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35439,127.0.97.61:45253,127.0.97.60:32831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:44.150870 13858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:44.151252 13858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:44.151423 13858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:44.158933 13858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:44.159118 13858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:44.159226 13858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:44.159332 13858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:44.159418 13858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:44.169653 13858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35439,127.0.97.61:45253,127.0.97.60:32831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:44.171149 13858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:44.172407 13858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:44.184511 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:44.184541 13866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:44.184492 13864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:44.185503 13858 server_base.cc:1034] running on GCE node
I20260923 20:56:44.186420 13858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:44.187937 13858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:44.189191 13858 hybrid_clock.cc:648] HybridClock initialized: now 1790197004189135 us; error 54 us; skew 500 ppm
I20260923 20:56:44.189584 13858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:44 dist-test-slave-plcg krb5kdc[13845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:44.198933 13858 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:56:44.200702 13858 webserver.cc:460] Webserver started at http://127.0.97.62:33209/ using document root <none> and password file <none>
I20260923 20:56:44.201483 13858 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:44.201656 13858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:44.202054 13858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:44.205566 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "dc5fc27a928644678f51d28d0d46ee64"
format_stamp: "Formatted at 2026-09-23 20:56:44 on dist-test-slave-plcg"
server_key: "6eb070edc0dc53ac48adcf8206636547"
server_key_iv: "20a88ae570cc57fd453e5df6dc1394ab"
server_key_version: "encryptionkey@0"
I20260923 20:56:44.206597 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "dc5fc27a928644678f51d28d0d46ee64"
format_stamp: "Formatted at 2026-09-23 20:56:44 on dist-test-slave-plcg"
server_key: "6eb070edc0dc53ac48adcf8206636547"
server_key_iv: "20a88ae570cc57fd453e5df6dc1394ab"
server_key_version: "encryptionkey@0"
I20260923 20:56:44.212170 13858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:56:44.215863 13873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:44.217343 13858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:56:44.217552 13858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "dc5fc27a928644678f51d28d0d46ee64"
format_stamp: "Formatted at 2026-09-23 20:56:44 on dist-test-slave-plcg"
server_key: "6eb070edc0dc53ac48adcf8206636547"
server_key_iv: "20a88ae570cc57fd453e5df6dc1394ab"
server_key_version: "encryptionkey@0"
I20260923 20:56:44.217804 13858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:44.240625 13858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:44.245360 13858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:44.245689 13858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:44.264918 13858 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35439
I20260923 20:56:44.264947 13924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35439 every 8 connection(s)
I20260923 20:56:44.266425 13858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:56:44.270545 13925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:44.272068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 13858
I20260923 20:56:44.272308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:56:44.272966 388 external_mini_cluster.cc:1428] Setting key 449a5ac7eaf679866287e5a82c494f6d
I20260923 20:56:44.277773 13925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 23 20:56:44 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:44.304201 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.281249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.0.97.62:35439)
0923 20:56:44.282214 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:44.282236 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:44.282674 (+ 438us) server_negotiation.cc:407] Connection header received
0923 20:56:44.284988 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:44.284994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:44.285267 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:44.285490 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:44.286878 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.288107 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.289586 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.290261 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.290484 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.296576 (+ 6092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:44.296586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:44.296602 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:44.296643 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:44.299714 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.300494 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.300500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.300503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.300652 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.301061 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.301072 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.301075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.301893 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:44.302133 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:44.302819 (+ 686us) server_negotiation.cc:299] Negotiation successful
0923 20:56:44.303135 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":698,"thread_start_us":461,"threads_started":1}
I20260923 20:56:44.304242 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.281249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35439 (local address 127.0.0.1:48208)
0923 20:56:44.282497 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.282533 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:44.284541 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:44.285500 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:44.285506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:44.286307 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:44.286657 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.286675 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.288283 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.288293 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.289425 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.289438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.290516 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.290523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.290700 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.291401 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:44.291431 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:44.296303 (+ 4872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:44.299889 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.299904 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.299918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.300336 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.300767 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.300774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.300777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.300940 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.302101 (+ 1161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:44.302108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:44.302333 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 20:56:44.302667 (+ 334us) client_negotiation.cc:240] Negotiation successful
0923 20:56:44.303350 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":107,"spinlock_wait_cycles":24320,"thread_start_us":810,"threads_started":1}
I20260923 20:56:44.307670 13925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:56:44.309028 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.308476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:40950)
0923 20:56:44.308705 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.308852 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 20:56:44.309410 13877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
I20260923 20:56:44.311394 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.310835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:40956)
0923 20:56:44.311056 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.311177 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:56:44.311641 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45253: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 20:56:44.324963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45253
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35439,127.0.97.61:45253,127.0.97.60:32831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:56:44.361042 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } attempt: 1
I20260923 20:56:44.362425 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.361889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:40968)
0923 20:56:44.362131 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.362259 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:56:44.363540 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.363118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:40974)
0923 20:56:44.363283 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.363370 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:56:44.363765 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45253: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
I20260923 20:56:44.432045 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } attempt: 2
I20260923 20:56:44.433394 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.432790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:40990)
0923 20:56:44.433094 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.433216 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 20:56:44.434638 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.434178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41000)
0923 20:56:44.434359 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.434454 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:56:44.434868 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45253: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
I20260923 20:56:44.526171 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } attempt: 3
I20260923 20:56:44.527602 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.526960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41012)
0923 20:56:44.527240 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.527374 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 20:56:44.528833 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.528292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41016)
0923 20:56:44.528496 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.528617 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:56:44.529052 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45253: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
W20260923 20:56:44.669288 13932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:44.669595 13932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:44.669763 13932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:56:44.672341 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } attempt: 4
I20260923 20:56:44.673658 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.673012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41020)
0923 20:56:44.673349 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.673471 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260923 20:56:44.674916 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.674456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41022)
0923 20:56:44.674640 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.674748 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 20:56:44.675169 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45253: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45253: connect: Connection refused (error 111)
W20260923 20:56:44.678277 13932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:44.678383 13932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:44.678431 13932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:44.678489 13932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:44.678534 13932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:44.689165 13932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35439,127.0.97.61:45253,127.0.97.60:32831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:44.690608 13932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:44.691890 13932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:44.704130 13937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:44.704085 13938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:44.704039 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:44.705188 13932 server_base.cc:1034] running on GCE node
I20260923 20:56:44.706135 13932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:44.707716 13932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:44.708950 13932 hybrid_clock.cc:648] HybridClock initialized: now 1790197004708894 us; error 43 us; skew 500 ppm
I20260923 20:56:44.709350 13932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:44 dist-test-slave-plcg krb5kdc[13845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:44.712940 13932 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 20:56:44.714958 13932 webserver.cc:460] Webserver started at http://127.0.97.61:36459/ using document root <none> and password file <none>
I20260923 20:56:44.715780 13932 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:44.715956 13932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:44.716329 13932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:44.720018 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data/instance:
uuid: "6c913fd32748489eb78dee65e4d7eeb5"
format_stamp: "Formatted at 2026-09-23 20:56:44 on dist-test-slave-plcg"
server_key: "58ce0d07bf1b4b9c58984a94f7cdde38"
server_key_iv: "c5653616814bfa69db47a9daeb0369e8"
server_key_version: "encryptionkey@0"
I20260923 20:56:44.720993 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/wal/instance:
uuid: "6c913fd32748489eb78dee65e4d7eeb5"
format_stamp: "Formatted at 2026-09-23 20:56:44 on dist-test-slave-plcg"
server_key: "58ce0d07bf1b4b9c58984a94f7cdde38"
server_key_iv: "c5653616814bfa69db47a9daeb0369e8"
server_key_version: "encryptionkey@0"
I20260923 20:56:44.727358 13932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 20:56:44.731261 13947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:44.732728 13932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:56:44.732950 13932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/wal
uuid: "6c913fd32748489eb78dee65e4d7eeb5"
format_stamp: "Formatted at 2026-09-23 20:56:44 on dist-test-slave-plcg"
server_key: "58ce0d07bf1b4b9c58984a94f7cdde38"
server_key_iv: "c5653616814bfa69db47a9daeb0369e8"
server_key_version: "encryptionkey@0"
I20260923 20:56:44.733183 13932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:44.752091 13932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:44.756975 13932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:44.757320 13932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:44.777621 13932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45253
I20260923 20:56:44.777674 13998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45253 every 8 connection(s)
I20260923 20:56:44.779119 13932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
I20260923 20:56:44.782979 13999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:44.788911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 13932
I20260923 20:56:44.789142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-1/wal/instance
I20260923 20:56:44.789585 388 external_mini_cluster.cc:1428] Setting key 72e4272d953161b672b260bedde7f412
I20260923 20:56:44.790199 13999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:56:44 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 20:56:44.815132 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.793121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.0.97.62:35439)
0923 20:56:44.793355 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:44.793361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:44.794758 (+ 1397us) server_negotiation.cc:407] Connection header received
0923 20:56:44.797365 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:44.797373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:44.797481 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:44.797605 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:44.799410 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.800581 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.802510 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.803335 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.803490 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.807413 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:44.807424 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:44.807433 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:44.807492 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:44.811335 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.812305 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.812310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.812312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.812449 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.812960 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.812964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.812967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.813363 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:44.813531 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:44.814528 (+ 997us) server_negotiation.cc:299] Negotiation successful
0923 20:56:44.814799 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:56:44.816236 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.793567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35439 (local address 127.0.0.1:48222)
0923 20:56:44.794553 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.794608 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:44.797043 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:44.797933 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:44.797943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:44.798545 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:44.799178 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.799198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.800818 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.800822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.802319 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.802335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.803593 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.803599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.803825 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.804600 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:44.804625 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:44.807062 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:44.811549 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.811558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.811574 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.812130 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.812581 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.812587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.812594 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.812804 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.813504 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:44.813513 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:44.814089 (+ 576us) client_negotiation.cc:769] Sending connection context
0923 20:56:44.814402 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 20:56:44.814969 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":415,"threads_started":1}
I20260923 20:56:44.818188 13999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:56:44 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:56:44.835925 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.819028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41028)
0923 20:56:44.819302 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.819326 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:44.819457 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:44.820763 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:44.820767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:44.821361 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:44.821588 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.821603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.823319 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.823324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.824785 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.824805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.826284 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.826338 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.826628 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.827358 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:44.827376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:44.829493 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:44.833184 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.833190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.833194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.833602 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.834094 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.834099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.834103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.834254 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.834901 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:44.834906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:44.835079 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 20:56:44.835228 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 20:56:44.835543 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 20:56:44.836047 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.819297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.0.97.61:45253)
0923 20:56:44.820152 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:44.820163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:44.820199 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:56:44.820264 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:44.820275 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:44.820563 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:44.820731 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:44.821808 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.823096 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.825012 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.826061 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.826238 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.829741 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:44.829755 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:44.829771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:44.829808 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:44.833012 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.833751 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.833761 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.833764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.833975 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.834371 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.834376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.834379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.834763 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:44.834992 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:44.835370 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 20:56:44.835667 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":633,"thread_start_us":470,"threads_started":1}
I20260923 20:56:44.838819 13999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:56:44.840013 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.839514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44100)
0923 20:56:44.839744 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.839854 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260923 20:56:44.840265 13950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
I20260923 20:56:44.842075 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.841601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44114)
0923 20:56:44.841774 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.841910 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 20:56:44.842315 13999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 20:56:44.846359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32831
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35439,127.0.97.61:45253,127.0.97.60:32831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:56:44.891708 13999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 1
I20260923 20:56:44.892925 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.892448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44120)
0923 20:56:44.892663 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.892772 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 20:56:44.894016 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.893594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44134)
0923 20:56:44.893750 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.893900 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:56:44.894269 13999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
I20260923 20:56:44.962504 13999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 2
I20260923 20:56:44.963846 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.963264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44142)
0923 20:56:44.963522 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.963650 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 20:56:44.964934 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.964537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44144)
0923 20:56:44.964693 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.964784 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:56:44.965199 13999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
I20260923 20:56:44.974428 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } attempt: 5
Sep 23 20:56:44 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:56:44.988356 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.975310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.97.61:45253)
0923 20:56:44.975506 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:44.975514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:44.975538 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:56:44.975757 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:44.975765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:44.975835 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:44.975946 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:44.976798 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.977488 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.978776 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.979259 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.979405 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.982675 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:44.982681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:44.982684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:44.982720 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:44.985628 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.986392 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.986399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.986407 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.986531 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:44.986936 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:44.986943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:44.986952 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:44.987235 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:44.987392 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:44.987814 (+ 422us) server_negotiation.cc:299] Negotiation successful
0923 20:56:44.988033 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:56:44.988372 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.975221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41038)
0923 20:56:44.975451 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.975472 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:44.975617 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:44.975981 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:44.975986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:44.976451 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:44.976639 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.976648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.977649 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.977660 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.978607 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:44.978619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:44.979423 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:44.979429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:44.979652 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:44.980317 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:44.980335 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:44.982431 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:44.985805 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.985811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.985836 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.986258 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.986661 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:44.986667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:44.986670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:44.986815 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:44.987357 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:44.987362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:44.987546 (+ 184us) client_negotiation.cc:769] Sending connection context
0923 20:56:44.987691 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 20:56:44.988010 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 20:56:44.989732 13925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:56:44.990818 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.990410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44154)
0923 20:56:44.990579 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.990674 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:56:44.991734 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:44.991390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44162)
0923 20:56:44.991530 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:44.991632 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:56:44.991962 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
I20260923 20:56:45.043200 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 1
I20260923 20:56:45.044616 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.043880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44176)
0923 20:56:45.044172 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.044350 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 20:56:45.045977 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.045436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44184)
0923 20:56:45.045633 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.045752 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:56:45.046257 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
I20260923 20:56:45.056473 13999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 3
I20260923 20:56:45.057746 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.057251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44190)
0923 20:56:45.057493 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.057586 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:56:45.058914 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.058452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44194)
0923 20:56:45.058707 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.058783 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260923 20:56:45.059155 13999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
I20260923 20:56:45.114521 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 2
I20260923 20:56:45.115906 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.115302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44206)
0923 20:56:45.115615 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.115730 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:56:45.116994 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.116629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44216)
0923 20:56:45.116792 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.116871 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:56:45.117264 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
W20260923 20:56:45.192679 14006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:45.193020 14006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:45.193186 14006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:45.200994 14006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:45.201129 14006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:45.201215 14006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:45.201306 14006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:45.201375 14006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:45.202440 13999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 4
I20260923 20:56:45.203754 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.203171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44226)
0923 20:56:45.203449 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.203561 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 20:56:45.204972 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.204566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44228)
0923 20:56:45.204737 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.204826 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 20:56:45.205197 13999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
I20260923 20:56:45.213366 14006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35439,127.0.97.61:45253,127.0.97.60:32831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:32831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:45.215114 14006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:45.216548 14006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 20:56:45.223526 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 3
I20260923 20:56:45.224761 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.224279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44230)
0923 20:56:45.224512 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.224601 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:56:45.226137 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.225685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44234)
0923 20:56:45.225924 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.226002 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260923 20:56:45.226460 13925 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:32831: Network error: Client connection negotiation failed: client connection to 127.0.97.60:32831: connect: Connection refused (error 111)
W20260923 20:56:45.231559 14011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:45.231482 14014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:45.231528 14012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:45.232244 14006 server_base.cc:1034] running on GCE node
I20260923 20:56:45.233088 14006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:45.234728 14006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:45.235967 14006 hybrid_clock.cc:648] HybridClock initialized: now 1790197005235916 us; error 47 us; skew 500 ppm
I20260923 20:56:45.236346 14006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:45.239861 14006 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 20:56:45.241577 14006 webserver.cc:460] Webserver started at http://127.0.97.60:32859/ using document root <none> and password file <none>
I20260923 20:56:45.242395 14006 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:45.242552 14006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:45.242872 14006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:45.246141 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data/instance:
uuid: "ac69f52ef5904ea0a5542add7ed1b2b5"
format_stamp: "Formatted at 2026-09-23 20:56:45 on dist-test-slave-plcg"
server_key: "7bad1cebd5b880b35580b9be6c077ca1"
server_key_iv: "51eafd44250ac6655c1dc63e0f7b4dc9"
server_key_version: "encryptionkey@0"
I20260923 20:56:45.247005 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/wal/instance:
uuid: "ac69f52ef5904ea0a5542add7ed1b2b5"
format_stamp: "Formatted at 2026-09-23 20:56:45 on dist-test-slave-plcg"
server_key: "7bad1cebd5b880b35580b9be6c077ca1"
server_key_iv: "51eafd44250ac6655c1dc63e0f7b4dc9"
server_key_version: "encryptionkey@0"
I20260923 20:56:45.252797 14006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:56:45.256613 14021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:45.258181 14006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:56:45.258400 14006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/wal
uuid: "ac69f52ef5904ea0a5542add7ed1b2b5"
format_stamp: "Formatted at 2026-09-23 20:56:45 on dist-test-slave-plcg"
server_key: "7bad1cebd5b880b35580b9be6c077ca1"
server_key_iv: "51eafd44250ac6655c1dc63e0f7b4dc9"
server_key_version: "encryptionkey@0"
I20260923 20:56:45.258607 14006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:45.297964 14006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:45.302670 14006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:45.302922 14006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:45.322544 14006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:32831
I20260923 20:56:45.322580 14072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:32831 every 8 connection(s)
I20260923 20:56:45.323987 14006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
I20260923 20:56:45.327812 14073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:45.330006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14006
I20260923 20:56:45.330205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790196790989944-388-0/minicluster-data/master-2/wal/instance
I20260923 20:56:45.330662 388 external_mini_cluster.cc:1428] Setting key 518736c1ff92aa997faa9394462d568b
I20260923 20:56:45.335520 14073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:45.355748 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.338583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.97.62:35439)
0923 20:56:45.339162 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.339168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.339676 (+ 508us) server_negotiation.cc:407] Connection header received
0923 20:56:45.339871 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.339875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.339952 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.340046 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.341314 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.342697 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.344339 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.345140 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.345315 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.349117 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.349128 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.349134 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.349179 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.352448 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.353284 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.353288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.353290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.353419 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.353862 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.353866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.353869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.354233 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.354373 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.355064 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.355296 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":261,"threads_started":1}
I20260923 20:56:45.356335 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.338591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35439 (local address 127.0.0.1:48234)
0923 20:56:45.339510 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.339561 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:45.339738 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:45.340137 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:45.340144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:45.340872 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:45.341137 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.341146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.342945 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.342951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.344149 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.344161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.345424 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.345431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.345567 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.346271 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:45.346307 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:45.348868 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:45.352587 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.352600 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.352615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.353157 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.353532 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.353537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.353540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.353708 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.354359 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:45.354367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:45.354557 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 20:56:45.354741 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 20:56:45.355235 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":190,"thread_start_us":480,"threads_started":1}
I20260923 20:56:45.356390 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.332749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.0.97.60:32831)
0923 20:56:45.333621 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.333633 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.333675 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:56:45.335392 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.335416 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.335847 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.336274 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.338263 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.340363 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.342216 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.342961 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.343185 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.348237 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.348247 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.348268 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.348305 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.351325 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.352283 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.352290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.352297 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.352455 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.352904 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.352909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.352912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.353694 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.353987 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.354544 (+ 557us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.355083 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":643,"thread_start_us":416,"threads_started":1}
I20260923 20:56:45.358114 14073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:45.373353 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.358915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.97.61:45253)
0923 20:56:45.359103 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.359109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.359130 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:56:45.359234 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.359239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.359313 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.359417 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.360362 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.361060 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.362456 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.362940 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.363080 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.366690 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.366695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.366699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.366734 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.370583 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.371428 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.371437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.371443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.371607 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.371978 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.371983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.371987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.372356 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.372529 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.372905 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.373084 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:56:45.373517 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.358807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45253 (local address 127.0.0.1:41042)
0923 20:56:45.358980 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.359000 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:45.359122 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:45.359439 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:45.359444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:45.359977 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:45.360210 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.360222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.361215 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.361220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.362275 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.362288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.363101 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.363108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.363369 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.364092 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:45.364115 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:45.366460 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:45.370751 (+ 4291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.370755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.370758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.371290 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.371726 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.371729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.371731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.371863 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.372482 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:45.372485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:45.372619 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 20:56:45.372801 (+ 182us) client_negotiation.cc:240] Negotiation successful
0923 20:56:45.373101 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:56:45.374706 14073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:56:45.375805 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.359865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.0.97.62:35439)
0923 20:56:45.360036 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.360040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.360064 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:56:45.360180 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.360184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.360258 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.360368 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.361556 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.362424 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.363979 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.364637 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.364775 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.369038 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.369044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.369049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.369096 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.372570 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.373454 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.373462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.373466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.373641 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.374188 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.374193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.374195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.374583 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.374876 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.375223 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.375465 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:56:45.388815 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.375641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.97.60:32831)
0923 20:56:45.375795 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.375800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.375816 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:56:45.375857 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.375860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.375956 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.376053 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.376966 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.377949 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.379086 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.379505 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.379631 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.382996 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.383002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.383006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.383045 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.386037 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.386669 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.386676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.386680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.386835 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.387239 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.387242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.387244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.387508 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.387640 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.388182 (+ 542us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.388405 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":52}
I20260923 20:56:45.388886 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.375408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44252)
0923 20:56:45.375574 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.375592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:45.375690 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:45.376098 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:45.376102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:45.376527 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:45.376701 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.376713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.378067 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.378075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.378939 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.378956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.379645 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.379650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.379892 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.380504 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:45.380532 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:45.382817 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:45.386178 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.386182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.386184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.386546 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.386948 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.386965 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.386968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.387130 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.387797 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:45.387803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:45.387941 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 20:56:45.388107 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 20:56:45.388559 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":162}
I20260923 20:56:45.392669 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.377648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.0.97.61:45253)
0923 20:56:45.377849 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.377855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.377874 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:56:45.378024 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.378030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.378112 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.378215 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.379389 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.380114 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.381356 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.382059 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.382191 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.386988 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.386995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.387002 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.387048 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.390060 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.390725 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.390733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.390736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.390877 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.391273 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.391279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.391282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.391557 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.391865 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.392114 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.392346 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:56:45.394091 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: Bootstrap starting.
I20260923 20:56:45.397729 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:56:45.397985 14073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:45.399071 14073 log.cc:824] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:45.402179 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: No bootstrap required, opened a new log
I20260923 20:56:45.403716 13925 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 4
I20260923 20:56:45.418516 14073 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:56:45.421183 14073 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:45.421414 14073 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac69f52ef5904ea0a5542add7ed1b2b5, State: Initialized, Role: FOLLOWER
I20260923 20:56:45.424002 14073 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.427809 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.404268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44276)
0923 20:56:45.404503 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.404522 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:45.404625 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:45.408248 (+ 3623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:45.408254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:45.408806 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:45.409027 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.409039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.412974 (+ 3935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.412981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.414502 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.414516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.415867 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.415874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.416113 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.416852 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:45.416877 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:45.419896 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:45.424592 (+ 4696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.424600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.424607 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.425125 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.425613 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.425618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.425621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.425854 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.426592 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:45.426598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:45.426789 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 20:56:45.426936 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:56:45.427375 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":107}
I20260923 20:56:45.428144 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.404873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.0.97.60:32831)
0923 20:56:45.407110 (+ 2237us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.407116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.407139 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:56:45.407192 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.407197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.407281 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.409232 (+ 1951us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.409552 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.411608 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.414944 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.415670 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.416474 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.420171 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.420177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.420182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.420229 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.424282 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.425305 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.425311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.425314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.425473 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.426024 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.426029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.426034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.426442 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.427211 (+ 769us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.427375 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.427634 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2098,"thread_start_us":1860,"threads_started":1}
I20260923 20:56:45.428984 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.403498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.0.97.62:35439)
0923 20:56:45.403711 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.403720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.404270 (+ 550us) server_negotiation.cc:407] Connection header received
0923 20:56:45.404543 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.404549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.404632 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.404711 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.407500 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.409013 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.411868 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.412805 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.412935 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.418232 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.418242 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.418247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.418306 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.423822 (+ 5516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.424756 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.424762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.424766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.424937 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.425462 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.425468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.425473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.425943 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.426231 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.426921 (+ 690us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.428526 (+ 1605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":13056}
I20260923 20:56:45.431813 14073 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:45.434798 13925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64: Bootstrap starting.
I20260923 20:56:45.437058 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.404575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.97.60:32831)
0923 20:56:45.404860 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.404871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.417406 (+ 12535us) server_negotiation.cc:407] Connection header received
0923 20:56:45.417881 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.417887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.417968 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.418042 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.419891 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.421091 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.423431 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.424273 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.424382 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.428126 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.428136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.428140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.428179 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.432603 (+ 4424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.433437 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.433445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.433448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.433617 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.434151 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.434202 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.434208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.434707 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.436278 (+ 1571us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.436428 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.436694 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260923 20:56:45.440151 13925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:45.440477 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.403251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.97.61:45253)
0923 20:56:45.403528 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.403563 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.414371 (+ 10808us) server_negotiation.cc:407] Connection header received
0923 20:56:45.414684 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.414691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.414783 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.414862 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.419085 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.420208 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.424770 (+ 4562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.425493 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.425606 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.433410 (+ 7804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.433417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.433422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.433506 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.437258 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.438157 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.438164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.438167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.438319 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.438756 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.438760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.438763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.439167 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.439421 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.439839 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.440104 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 20:56:45.442075 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.442744 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:56:45.444129 13925 log.cc:824] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:45.450644 13925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64: No bootstrap required, opened a new log
W20260923 20:56:45.458557 14100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:56:45.458819 14100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:56:45.462404 13925 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.462947 13925 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:45.463115 13925 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5fc27a928644678f51d28d0d46ee64, State: Initialized, Role: FOLLOWER
I20260923 20:56:45.464524 13925 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.465978 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.466490 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:56:45.467361 13925 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:45.454819 14006 master_runner.cc:186] Error getting master registration for 127.0.97.62:35439: OK, instance_id { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" instance_seqno: 1790197004259345 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:56:45.468274 14006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 20:56:45.479424 14113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:56:45.479638 14113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:56:45.476926 13858 master_runner.cc:186] Error getting master registration for 127.0.97.61:45253: OK, instance_id { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" instance_seqno: 1790197004771944 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:56:45.490232 13858 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 20:56:45.498936 13879 tablet.cc:2369] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:56:45.504715 13999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } attempt: 5
Sep 23 20:56:45 dist-test-slave-plcg krb5kdc[13845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:56:45.525117 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.505697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.0.97.60:32831)
0923 20:56:45.505919 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:45.505925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:45.505946 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:56:45.506006 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:45.506012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:45.506116 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:45.506230 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:45.507449 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.508676 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.510530 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.511241 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.511382 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.515862 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:45.515873 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:45.515883 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:45.515930 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:45.521461 (+ 5531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.522601 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.522607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.522610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.522797 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:45.523343 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:45.523348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:45.523352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:45.523746 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:45.523917 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:45.524425 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 20:56:45.524695 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:56:45.525220 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:45.505329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:32831 (local address 127.0.0.1:44280)
0923 20:56:45.505618 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:45.505641 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:45.505760 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:45.506430 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:45.506435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:45.507011 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:45.507239 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.507252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.508865 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.508872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.510353 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:45.510372 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:45.511434 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:45.511441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:45.511668 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:45.512453 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:45.512478 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:45.515558 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:45.521663 (+ 6105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.521670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.521674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.522405 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.522942 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:45.522948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:45.522952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:45.523179 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:45.523913 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:45.523921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:45.524110 (+ 189us) client_negotiation.cc:769] Sending connection context
0923 20:56:45.524286 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 20:56:45.524815 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260923 20:56:45.533113 13999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: Bootstrap starting.
I20260923 20:56:45.537763 13999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:45.539366 13999 log.cc:824] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:45.543205 13999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: No bootstrap required, opened a new log
W20260923 20:56:45.555087 14027 tablet.cc:2369] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:56:45.556274 13999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.556761 13999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:45.556886 13999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c913fd32748489eb78dee65e4d7eeb5, State: Initialized, Role: FOLLOWER
I20260923 20:56:45.557636 13999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.558730 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.559289 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:56:45.560235 13999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:56:45.584950 14127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:56:45.585266 14127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:56:45.585671 13932 master_runner.cc:263] No leader master found from master 6c913fd32748489eb78dee65e4d7eeb5
I20260923 20:56:45.585803 13932 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:56:45.587190 14102 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:56:45.587528 14102 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.589211 14102 leader_election.cc:288] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c913fd32748489eb78dee65e4d7eeb5 (127.0.97.61:45253), ac69f52ef5904ea0a5542add7ed1b2b5 (127.0.97.60:32831)
I20260923 20:56:45.589970 13974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5fc27a928644678f51d28d0d46ee64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c913fd32748489eb78dee65e4d7eeb5" is_pre_election: true
I20260923 20:56:45.590257 14048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5fc27a928644678f51d28d0d46ee64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" is_pre_election: true
I20260923 20:56:45.590574 13974 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5fc27a928644678f51d28d0d46ee64 in term 0.
I20260923 20:56:45.590816 14048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5fc27a928644678f51d28d0d46ee64 in term 0.
I20260923 20:56:45.591295 13877 leader_election.cc:302] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c913fd32748489eb78dee65e4d7eeb5, dc5fc27a928644678f51d28d0d46ee64; no voters:
I20260923 20:56:45.592217 14102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:56:45.592388 14102 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:56:45.592518 14102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:45.596100 14102 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.597548 13974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5fc27a928644678f51d28d0d46ee64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c913fd32748489eb78dee65e4d7eeb5"
I20260923 20:56:45.597798 13974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:45.601410 13974 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc5fc27a928644678f51d28d0d46ee64 in term 1.
I20260923 20:56:45.602545 14048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5fc27a928644678f51d28d0d46ee64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5"
I20260923 20:56:45.602785 14048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:45.603711 13877 leader_election.cc:302] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c913fd32748489eb78dee65e4d7eeb5, dc5fc27a928644678f51d28d0d46ee64; no voters:
I20260923 20:56:45.606393 14048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc5fc27a928644678f51d28d0d46ee64 in term 1.
I20260923 20:56:45.607137 14102 leader_election.cc:288] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [CANDIDATE]: Term 1 election: Requested vote from peers 6c913fd32748489eb78dee65e4d7eeb5 (127.0.97.61:45253), ac69f52ef5904ea0a5542add7ed1b2b5 (127.0.97.60:32831)
I20260923 20:56:45.607429 14102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:56:45.607656 14102 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [term 1 LEADER]: Becoming Leader. State: Replica: dc5fc27a928644678f51d28d0d46ee64, State: Running, Role: LEADER
I20260923 20:56:45.608322 14102 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } }
I20260923 20:56:45.621241 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5fc27a928644678f51d28d0d46ee64. Latest consensus state: current_term: 1 leader_uuid: "dc5fc27a928644678f51d28d0d46ee64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.622175 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:45.629187 14131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:56:45.631714 14131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:56:45.648782 14048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [term 1 FOLLOWER]: Refusing update from remote peer dc5fc27a928644678f51d28d0d46ee64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:45.650074 14102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:45.657868 13974 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [term 1 FOLLOWER]: Refusing update from remote peer dc5fc27a928644678f51d28d0d46ee64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:45.659034 14102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:45.684650 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5fc27a928644678f51d28d0d46ee64. Latest consensus state: current_term: 1 leader_uuid: "dc5fc27a928644678f51d28d0d46ee64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.684965 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:56:45.688618 14132 mvcc.cc:205] Tried to move back new op lower bound from 7332646935123533824 to 7332646934977138688. Current Snapshot: MvccSnapshot[applied={T|T < 7332646935123533824}]
I20260923 20:56:45.701028 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc5fc27a928644678f51d28d0d46ee64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.701393 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:45.705832 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5fc27a928644678f51d28d0d46ee64. Latest consensus state: current_term: 1 leader_uuid: "dc5fc27a928644678f51d28d0d46ee64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.706203 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:56:45.712984 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc5fc27a928644678f51d28d0d46ee64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.713445 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:45.714069 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc5fc27a928644678f51d28d0d46ee64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.714339 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:56:45.714293 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc5fc27a928644678f51d28d0d46ee64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5fc27a928644678f51d28d0d46ee64" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35439 } } peers { permanent_uuid: "6c913fd32748489eb78dee65e4d7eeb5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45253 } } peers { permanent_uuid: "ac69f52ef5904ea0a5542add7ed1b2b5" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 32831 } } }
I20260923 20:56:45.714602 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:56:45.720023 14131 catalog_manager.cc:1347] Generated new cluster ID: 854f954954c44d2eae4ec04502e7fa96
I20260923 20:56:45.720148 14131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:56:45.748402 14133 mvcc.cc:205] Tried to move back new op lower bound from 7332646935123533824 to 7332646934977138688. Current Snapshot: MvccSnapshot[applied={T|T < 7332646935123533824 or (T in {7332646935123533824})}]
I20260923 20:56:45.751950 14131 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:56:45.759338 14131 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:56:45.780501 14131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc5fc27a928644678f51d28d0d46ee64: Generated new TSK 0
I20260923 20:56:45.781723 14131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:56:46.461177 14100 catalog_manager.cc:1259] Loaded cluster ID: 854f954954c44d2eae4ec04502e7fa96
I20260923 20:56:46.461359 14100 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: loading cluster ID for follower catalog manager: success
I20260923 20:56:46.464347 14100 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: acquiring CA information for follower catalog manager: success
I20260923 20:56:46.466480 14100 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac69f52ef5904ea0a5542add7ed1b2b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:56:46.587294 14127 catalog_manager.cc:1259] Loaded cluster ID: 854f954954c44d2eae4ec04502e7fa96
I20260923 20:56:46.587400 14127 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: loading cluster ID for follower catalog manager: success
I20260923 20:56:46.590073 14127 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: acquiring CA information for follower catalog manager: success
I20260923 20:56:46.591991 14127 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c913fd32748489eb78dee65e4d7eeb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:56:46.932572 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 13858
I20260923 20:56:46.934077 13920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:47.046779 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 13858
W20260923 20:56:47.068522 13951 connection.cc:535] server connection from 127.0.0.1:41038 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:56:47.068920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 13932
I20260923 20:56:47.070142 13994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:47.188786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 13932
I20260923 20:56:47.215622 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14006
I20260923 20:56:47.216759 14068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:47.325544 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14006
2026-09-23T20:56:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4882 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:56:47 dist-test-slave-plcg krb5kdc[14151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:56:47 dist-test-slave-plcg krb5kdc[14151](info): set up 2 sockets
Sep 23 20:56:47 dist-test-slave-plcg krb5kdc[14151](info): commencing operation
krb5kdc: starting...
W20260923 20:56:48.430490 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:56:48 dist-test-slave-plcg krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:56:48.681939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37991
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:49.025251 14164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:49.025640 14164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:49.025846 14164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:49.033602 14164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:49.033797 14164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:49.033921 14164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:49.034030 14164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:49.034124 14164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:49.044507 14164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:49.046020 14164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:49.047390 14164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:49.060317 14169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:49.060226 14172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:49.060287 14170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:49.061169 14164 server_base.cc:1034] running on GCE node
I20260923 20:56:49.062162 14164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:49.063761 14164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:49.065030 14164 hybrid_clock.cc:648] HybridClock initialized: now 1790197009064988 us; error 63 us; skew 500 ppm
I20260923 20:56:49.065419 14164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:49 dist-test-slave-plcg krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:49.068759 14164 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:56:49.070662 14164 webserver.cc:460] Webserver started at http://127.0.97.62:35845/ using document root <none> and password file <none>
I20260923 20:56:49.071429 14164 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:49.071592 14164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:49.071930 14164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:49.075251 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc"
format_stamp: "Formatted at 2026-09-23 20:56:49 on dist-test-slave-plcg"
server_key: "6cd8e91c7352ebc884dcb7a3e7079507"
server_key_iv: "7c1ce1a9d6ac7fde8149f5b9f3b8c5ce"
server_key_version: "encryptionkey@0"
I20260923 20:56:49.076104 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc"
format_stamp: "Formatted at 2026-09-23 20:56:49 on dist-test-slave-plcg"
server_key: "6cd8e91c7352ebc884dcb7a3e7079507"
server_key_iv: "7c1ce1a9d6ac7fde8149f5b9f3b8c5ce"
server_key_version: "encryptionkey@0"
I20260923 20:56:49.082362 14164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 20:56:49.086864 14179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:49.088390 14164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:56:49.088616 14164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc"
format_stamp: "Formatted at 2026-09-23 20:56:49 on dist-test-slave-plcg"
server_key: "6cd8e91c7352ebc884dcb7a3e7079507"
server_key_iv: "7c1ce1a9d6ac7fde8149f5b9f3b8c5ce"
server_key_version: "encryptionkey@0"
I20260923 20:56:49.088867 14164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:49.110318 14164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:49.115830 14164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:49.116175 14164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:49.136649 14164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37991
I20260923 20:56:49.136682 14230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37991 every 8 connection(s)
I20260923 20:56:49.138247 14164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:56:49.142283 14231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:49.147692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14164
I20260923 20:56:49.147928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:56:49.148729 388 external_mini_cluster.cc:1428] Setting key 46f2c3365978c1e2aef69d89cd2dbf2d
I20260923 20:56:49.154789 14231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc: Bootstrap starting.
I20260923 20:56:49.158955 14231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:49.160395 14231 log.cc:824] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:49.164626 14231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc: No bootstrap required, opened a new log
Sep 23 20:56:49 dist-test-slave-plcg krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:49.173326 14231 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260923 20:56:49.173791 14231 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:49.173964 14231 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeaf0fe2eeaf43d186bd87b9eea560cc, State: Initialized, Role: FOLLOWER
I20260923 20:56:49.174751 14231 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260923 20:56:49.175057 14231 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:56:49.175206 14231 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:56:49.175408 14231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:49.176460 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:49.150926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.0.97.62:37991)
0923 20:56:49.151871 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:49.151884 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:49.151932 (+ 48us) server_negotiation.cc:407] Connection header received
0923 20:56:49.153754 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:49.153780 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:49.154324 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:49.154698 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:49.156708 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:49.158152 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:49.160183 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:49.161906 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:49.162157 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:49.167585 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:49.167596 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:49.167618 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:49.167689 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:49.171222 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:49.172132 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:49.172150 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:49.172157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:49.172358 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:49.172810 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:49.172815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:49.172820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:49.173435 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:49.173760 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:49.174429 (+ 669us) server_negotiation.cc:299] Negotiation successful
0923 20:56:49.174927 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":452,"threads_started":1}
I20260923 20:56:49.178815 14231 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260923 20:56:49.179448 14231 leader_election.cc:302] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeaf0fe2eeaf43d186bd87b9eea560cc; no voters:
I20260923 20:56:49.180119 14231 leader_election.cc:288] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:56:49.180296 14236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:56:49.181214 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:56:49.182616 14231 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:49.185801 14236 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [term 1 LEADER]: Becoming Leader. State: Replica: aeaf0fe2eeaf43d186bd87b9eea560cc, State: Running, Role: LEADER
I20260923 20:56:49.186504 14236 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } }
I20260923 20:56:49.189694 14237 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } } }
I20260923 20:56:49.190186 14237 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:49.190667 14238 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeaf0fe2eeaf43d186bd87b9eea560cc. Latest consensus state: current_term: 1 leader_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaf0fe2eeaf43d186bd87b9eea560cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37991 } } }
I20260923 20:56:49.190972 14238 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:49.193563 14245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:56:49.198388 14245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:56:49.207624 14245 catalog_manager.cc:1347] Generated new cluster ID: 27fce231b3424290a92bdc603a8c3eed
I20260923 20:56:49.207772 14245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:56:49.221632 14245 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:56:49.223104 14245 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:56:49.231998 14245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeaf0fe2eeaf43d186bd87b9eea560cc: Generated new TSK 0
I20260923 20:56:49.233191 14245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:56:49.245280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14164
I20260923 20:56:49.246747 14226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:49.343364 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14164
2026-09-23T20:56:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2013 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:56:49 dist-test-slave-plcg krb5kdc[14259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:56:49 dist-test-slave-plcg krb5kdc[14259](info): set up 2 sockets
Sep 23 20:56:49 dist-test-slave-plcg krb5kdc[14259](info): commencing operation
krb5kdc: starting...
W20260923 20:56:50.478448 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:56:50 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:56:50.727857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:51.076324 14272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:51.076736 14272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:51.076905 14272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:51.084599 14272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:51.084786 14272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:51.084899 14272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:51.085005 14272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:51.085090 14272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:51.095417 14272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:51.096901 14272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:51.098304 14272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:51.110991 14278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:51.111069 14280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:51.110954 14277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:51.112354 14272 server_base.cc:1034] running on GCE node
I20260923 20:56:51.113202 14272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:51.114851 14272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:51.116117 14272 hybrid_clock.cc:648] HybridClock initialized: now 1790197011116075 us; error 62 us; skew 500 ppm
I20260923 20:56:51.116603 14272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:51 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:51.119904 14272 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:56:51.121628 14272 webserver.cc:460] Webserver started at http://127.0.97.62:36371/ using document root <none> and password file <none>
I20260923 20:56:51.122457 14272 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:51.122623 14272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:51.122949 14272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:51.126216 14272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "64a62a97b099eeb3f8b49412024e282c"
server_key_iv: "79a768450390756ebf331fac3e37b09b"
server_key_version: "encryptionkey@0"
I20260923 20:56:51.127028 14272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "64a62a97b099eeb3f8b49412024e282c"
server_key_iv: "79a768450390756ebf331fac3e37b09b"
server_key_version: "encryptionkey@0"
I20260923 20:56:51.132655 14272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:56:51.136693 14287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:51.138424 14272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:56:51.138643 14272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "64a62a97b099eeb3f8b49412024e282c"
server_key_iv: "79a768450390756ebf331fac3e37b09b"
server_key_version: "encryptionkey@0"
I20260923 20:56:51.138880 14272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:51.162258 14272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:51.166846 14272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:51.167172 14272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:51.187417 14272 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42989
I20260923 20:56:51.187434 14338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42989 every 8 connection(s)
I20260923 20:56:51.188922 14272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:56:51.192775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14272
I20260923 20:56:51.193009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:56:51.193162 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:51.193861 388 external_mini_cluster.cc:1428] Setting key 4e8c00bd9ab3c499d29ebe3828640206
I20260923 20:56:51.208395 14339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Bootstrap starting.
Sep 23 20:56:51 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:51.212327 14339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:51.213706 14339 log.cc:824] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:51.217095 14339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: No bootstrap required, opened a new log
I20260923 20:56:51.219987 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:51.196161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.0.97.62:42989)
0923 20:56:51.197146 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:51.197159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:51.197205 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:56:51.199761 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:51.199790 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:51.200261 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:51.200637 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:51.202539 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:51.203851 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:51.205957 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:51.206755 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:51.206920 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:51.211817 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:51.211826 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:51.211843 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:51.211891 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:51.215358 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:51.216260 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:51.216267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:51.216274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:51.216456 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:51.216908 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:51.216914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:51.216919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:51.217552 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:51.217946 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:51.218399 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 20:56:51.218873 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":462,"threads_started":1}
I20260923 20:56:51.225803 14339 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:51.226415 14339 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:51.226560 14339 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b8a1b3a2a24ff4ab94dc3f9157e6b9, State: Initialized, Role: FOLLOWER
I20260923 20:56:51.227257 14339 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:51.227528 14339 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:56:51.227696 14339 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:56:51.227874 14339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:51.230638 14339 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:51.231210 14339 leader_election.cc:302] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b8a1b3a2a24ff4ab94dc3f9157e6b9; no voters:
I20260923 20:56:51.231848 14339 leader_election.cc:288] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:56:51.232023 14344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:56:51.232839 14344 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 LEADER]: Becoming Leader. State: Replica: e4b8a1b3a2a24ff4ab94dc3f9157e6b9, State: Running, Role: LEADER
I20260923 20:56:51.233400 14344 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:51.234248 14339 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:51.237119 14346 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b8a1b3a2a24ff4ab94dc3f9157e6b9. Latest consensus state: current_term: 1 leader_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } } }
I20260923 20:56:51.237602 14346 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:51.237215 14345 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } } }
I20260923 20:56:51.237879 14345 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:51.239006 14353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:56:51.245100 14353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:56:51.255170 14353 catalog_manager.cc:1347] Generated new cluster ID: e0736177fbc749598462f341fc66d2e0
I20260923 20:56:51.255307 14353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:56:51.277357 14353 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:56:51.278878 14353 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:56:51.292520 14353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Generated new TSK 0
I20260923 20:56:51.293481 14353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:56:51.362560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42989
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:51.728873 14365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:51.729183 14365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:51.729281 14365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:51.737111 14365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:51.737242 14365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:51.737362 14365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:56:51.748073 14365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:51.749472 14365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:51.750979 14365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:51.765170 14370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:51.765228 14373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:51.765134 14371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:51.766003 14365 server_base.cc:1034] running on GCE node
I20260923 20:56:51.766724 14365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:51.767779 14365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:51.769034 14365 hybrid_clock.cc:648] HybridClock initialized: now 1790197011768962 us; error 64 us; skew 500 ppm
I20260923 20:56:51.769397 14365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:51 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:51.772835 14365 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:56:51.774573 14365 webserver.cc:460] Webserver started at http://127.0.97.1:43869/ using document root <none> and password file <none>
I20260923 20:56:51.775336 14365 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:51.775502 14365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:51.775837 14365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:51.779076 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "a43361c9bb7c408b93c7923859aab40f"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "1a45ab27b0034a1aa8fa41adc2087f53"
server_key_iv: "259e0559881a73aaf35430f876b3b974"
server_key_version: "encryptionkey@0"
I20260923 20:56:51.780019 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a43361c9bb7c408b93c7923859aab40f"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "1a45ab27b0034a1aa8fa41adc2087f53"
server_key_iv: "259e0559881a73aaf35430f876b3b974"
server_key_version: "encryptionkey@0"
I20260923 20:56:51.785647 14365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:56:51.789603 14380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:51.791127 14365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:56:51.791359 14365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "a43361c9bb7c408b93c7923859aab40f"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "1a45ab27b0034a1aa8fa41adc2087f53"
server_key_iv: "259e0559881a73aaf35430f876b3b974"
server_key_version: "encryptionkey@0"
I20260923 20:56:51.791595 14365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:51.811607 14365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:51.816545 14365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:51.816887 14365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:51.818058 14365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:51.819561 14365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:51.819685 14365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:51.819859 14365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:51.819938 14365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:51.856534 14365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39007
I20260923 20:56:51.856568 14492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39007 every 8 connection(s)
I20260923 20:56:51.858093 14365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:56:51.860987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14365
I20260923 20:56:51.861231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:56:51.862129 388 external_mini_cluster.cc:1428] Setting key 306f810d9a29603082d06b87e8225579
Sep 23 20:56:51 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:51.883271 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:51.861008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46589 (local address 127.0.97.62:42989)
0923 20:56:51.863178 (+ 2170us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:51.863185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:51.863451 (+ 266us) server_negotiation.cc:407] Connection header received
0923 20:56:51.865503 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:51.865518 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:51.865620 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:51.865751 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:51.867548 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:51.868275 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:51.870107 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:51.870531 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:51.870695 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:51.875260 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:51.875271 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:51.875279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:51.875339 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:51.879201 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:51.880171 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:51.880180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:51.880187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:51.880356 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:51.880983 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:51.880991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:51.880998 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:51.881432 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:51.881608 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:51.882488 (+ 880us) server_negotiation.cc:299] Negotiation successful
0923 20:56:51.882781 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2034,"thread_start_us":1925,"threads_started":1}
I20260923 20:56:51.884454 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:51.861713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:46589)
0923 20:56:51.863281 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:51.863333 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:51.865193 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:51.866066 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:51.866076 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:51.866708 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:51.867327 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:51.867346 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:51.868483 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:51.868488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:51.869902 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:51.869920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:51.870818 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:51.870826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:51.871096 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:51.871914 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:51.871953 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:51.874917 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:51.879441 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:51.879451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:51.879478 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:51.879983 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:51.880586 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:51.880594 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:51.880599 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:51.880822 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:51.881586 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:51.881593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:51.882047 (+ 454us) client_negotiation.cc:769] Sending connection context
0923 20:56:51.882295 (+ 248us) client_negotiation.cc:240] Negotiation successful
0923 20:56:51.882793 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":494,"threads_started":1}
I20260923 20:56:51.886559 14493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42989
I20260923 20:56:51.886952 14493 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:56:51.888043 14493 heartbeater.cc:511] Master 127.0.97.62:42989 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 20:56:51.891209 14304 ts_manager.cc:194] Registered new tserver with Master: a43361c9bb7c408b93c7923859aab40f (127.0.97.1:39007)
I20260923 20:56:51.892755 14304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:56:51.921618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42989
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:52.267118 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:52.267437 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:52.267540 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:52.275122 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:52.275274 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:52.275393 14499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:56:52.286044 14499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:52.287352 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:52.288669 14499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:52.303432 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:52.303378 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:52.303354 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:52.304354 14499 server_base.cc:1034] running on GCE node
I20260923 20:56:52.305116 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:52.306262 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:52.307538 14499 hybrid_clock.cc:648] HybridClock initialized: now 1790197012307483 us; error 65 us; skew 500 ppm
I20260923 20:56:52.307907 14499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:52 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:52.312182 14499 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:56:52.313987 14499 webserver.cc:460] Webserver started at http://127.0.97.2:38381/ using document root <none> and password file <none>
I20260923 20:56:52.314781 14499 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:52.314944 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:52.315289 14499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:52.318550 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "696c6aefbb924d28908dfae3dda3e317"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "6f8f8f76941fb5dfa4b72f08720525ef"
server_key_iv: "9417ebec7c753d25be1f286ce57f8a4d"
server_key_version: "encryptionkey@0"
I20260923 20:56:52.319434 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "696c6aefbb924d28908dfae3dda3e317"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "6f8f8f76941fb5dfa4b72f08720525ef"
server_key_iv: "9417ebec7c753d25be1f286ce57f8a4d"
server_key_version: "encryptionkey@0"
I20260923 20:56:52.325600 14499 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 20:56:52.329596 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:52.331506 14499 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:56:52.331745 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "696c6aefbb924d28908dfae3dda3e317"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "6f8f8f76941fb5dfa4b72f08720525ef"
server_key_iv: "9417ebec7c753d25be1f286ce57f8a4d"
server_key_version: "encryptionkey@0"
I20260923 20:56:52.331990 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:52.346562 14499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:52.351342 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:52.351676 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:52.352749 14499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:52.354215 14499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:52.354333 14499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:52.354491 14499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:52.354571 14499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:52.391119 14499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45919
I20260923 20:56:52.391168 14626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45919 every 8 connection(s)
I20260923 20:56:52.392486 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:56:52.396265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14499
I20260923 20:56:52.396461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:56:52.396910 388 external_mini_cluster.cc:1428] Setting key 45a5a55cbe359ff58e9d0522582f0fc5
Sep 23 20:56:52 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:52.418668 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:52.395539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42961 (local address 127.0.97.62:42989)
0923 20:56:52.396131 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:52.396137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:52.397364 (+ 1227us) server_negotiation.cc:407] Connection header received
0923 20:56:52.399626 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:52.399633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:52.399724 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:52.399857 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:52.401508 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.402734 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.405166 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.405988 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.406164 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:52.411384 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:52.411390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:52.411393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:52.411437 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:52.415034 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:52.415969 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:52.415978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:52.415982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:52.416139 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:52.416689 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:52.416694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:52.416697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:52.416997 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:52.417193 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:52.418001 (+ 808us) server_negotiation.cc:299] Negotiation successful
0923 20:56:52.418281 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":273,"threads_started":1}
I20260923 20:56:52.419744 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:52.396187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:42961)
0923 20:56:52.397145 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:52.397206 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:52.399303 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:52.400081 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:52.400091 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:52.400667 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:52.401283 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:52.401302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.402980 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.402985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:52.404958 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:52.404976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.406299 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.406311 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:52.406710 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:52.407620 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:52.407660 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:52.411058 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:52.415272 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:52.415287 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:52.415310 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:52.415807 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:52.416308 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:52.416317 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:52.416325 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:52.416548 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:52.417163 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:52.417172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:52.417577 (+ 405us) client_negotiation.cc:769] Sending connection context
0923 20:56:52.417916 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 20:56:52.418489 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":401,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 20:56:52.421473 14627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42989
I20260923 20:56:52.422091 14627 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:52.422960 14627 heartbeater.cc:511] Master 127.0.97.62:42989 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 20:56:52.424736 14304 ts_manager.cc:194] Registered new tserver with Master: 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919)
I20260923 20:56:52.425602 14304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:56:52.455096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42989
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:52.794386 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:52.794708 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:52.794844 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:52.802696 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:52.802834 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:52.803038 14633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:56:52.814209 14633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:52.815639 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:52.817061 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:52.832286 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:52.832279 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:52.832306 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:52.833273 14633 server_base.cc:1034] running on GCE node
I20260923 20:56:52.834055 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:52.835263 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:52.836585 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790197012836524 us; error 58 us; skew 500 ppm
I20260923 20:56:52.837056 14633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:52 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:52.840749 14633 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:56:52.842626 14633 webserver.cc:460] Webserver started at http://127.0.97.3:44133/ using document root <none> and password file <none>
I20260923 20:56:52.843559 14633 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:52.843735 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:52.844194 14633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:56:52.847509 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "45c15a1e7d134833a6002401f2d4fbbb"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "fdc1e68d74cbe97cd63e8426ad29db3e"
server_key_iv: "e82ae5b2fc75b7d8bb40b0a8f8e4c292"
server_key_version: "encryptionkey@0"
I20260923 20:56:52.848430 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "45c15a1e7d134833a6002401f2d4fbbb"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "fdc1e68d74cbe97cd63e8426ad29db3e"
server_key_iv: "e82ae5b2fc75b7d8bb40b0a8f8e4c292"
server_key_version: "encryptionkey@0"
I20260923 20:56:52.854344 14633 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:56:52.859258 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:52.861086 14633 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:56:52.861301 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "45c15a1e7d134833a6002401f2d4fbbb"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "fdc1e68d74cbe97cd63e8426ad29db3e"
server_key_iv: "e82ae5b2fc75b7d8bb40b0a8f8e4c292"
server_key_version: "encryptionkey@0"
I20260923 20:56:52.861572 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:52.886832 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:52.891341 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:52.891667 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:52.892709 14633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:52.894229 14633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:52.894337 14633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:52.894440 14633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:52.894485 14633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:52.896294 14493 heartbeater.cc:503] Master 127.0.97.62:42989 was elected leader, sending a full tablet report...
I20260923 20:56:52.930902 14633 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42851
I20260923 20:56:52.931031 14760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42851 every 8 connection(s)
I20260923 20:56:52.932349 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:56:52.940055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14633
I20260923 20:56:52.940217 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:56:52.940694 388 external_mini_cluster.cc:1428] Setting key d7ebcca75ee1c356fc14ae0c8703f114
Sep 23 20:56:52 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:52.955825 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:52.935320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54485 (local address 127.0.97.62:42989)
0923 20:56:52.935854 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:52.935859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:52.937085 (+ 1226us) server_negotiation.cc:407] Connection header received
0923 20:56:52.939266 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:52.939270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:52.939379 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:52.939483 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:52.941584 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.942809 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.945137 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.945540 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.945640 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:52.949326 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:52.949331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:52.949335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:52.949373 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:52.952154 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:52.953116 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:52.953120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:52.953123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:52.953260 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:52.953735 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:52.953738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:52.953740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:52.954100 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:52.954224 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:52.955170 (+ 946us) server_negotiation.cc:299] Negotiation successful
0923 20:56:52.955430 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":267,"threads_started":1}
I20260923 20:56:52.957099 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:52.935982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:54485)
0923 20:56:52.936890 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:52.936937 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:52.938906 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:52.939768 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:52.939777 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:52.940516 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:52.941353 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:52.941383 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.943045 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.943049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:52.944980 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:52.944996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.945773 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.945779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:52.946090 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:52.946758 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:52.946783 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:52.949083 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:52.952390 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:52.952399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:52.952414 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:52.952972 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:52.953378 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:52.953381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:52.953383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:52.953619 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:52.954230 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:52.954237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:52.954672 (+ 435us) client_negotiation.cc:769] Sending connection context
0923 20:56:52.955039 (+ 367us) client_negotiation.cc:240] Negotiation successful
0923 20:56:52.955653 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":360,"threads_started":1}
I20260923 20:56:52.958751 14761 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42989
I20260923 20:56:52.959121 14761 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:52.959942 14761 heartbeater.cc:511] Master 127.0.97.62:42989 requested a full tablet report, sending...
I20260923 20:56:52.961668 14304 ts_manager.cc:194] Registered new tserver with Master: 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
I20260923 20:56:52.962548 14304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54485
I20260923 20:56:52.970255 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:56:52.989508 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:52.975306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.0.97.62:42989)
0923 20:56:52.975738 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:52.975751 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:52.976520 (+ 769us) server_negotiation.cc:407] Connection header received
0923 20:56:52.976767 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:52.976771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:52.976858 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:52.976971 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:52.978594 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.979333 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.980767 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:52.981211 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:52.981368 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:52.983951 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:52.983956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:52.983959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:52.984001 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:52.986584 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:52.987367 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:52.987374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:52.987377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:52.987512 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:52.987926 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:52.987932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:52.987935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:52.988292 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:52.988563 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:52.988899 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 20:56:52.989155 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260923 20:56:52.994302 14304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:56:53.001634 14304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:56:53.030825 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.015146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45919 (local address 127.0.0.1:57546)
0923 20:56:53.016232 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:53.016293 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:53.016476 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:53.017785 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:53.017797 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:53.017870 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:53.018436 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.018455 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.021232 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.021242 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.024651 (+ 3409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.024678 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.029209 (+ 4531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.029221 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.029605 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.029686 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 20:56:53.029938 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 20:56:53.030256 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":605,"threads_started":1}
I20260923 20:56:53.032528 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.015339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.0.97.2:45919)
0923 20:56:53.016390 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.016399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.016431 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:56:53.016788 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.016802 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.017149 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.017410 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:53.018735 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.020955 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.024964 (+ 4009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.028494 (+ 3530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.028691 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.028981 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.031699 (+ 2718us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.031990 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":1214,"threads_started":1}
I20260923 20:56:53.038213 14562 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbd42a751a940249080cb1f204e6c5c (DEFAULT_TABLE table=test-table [id=63c4b6e20b604f86951b47042a7ad259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:53.040043 14562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbd42a751a940249080cb1f204e6c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:53.042208 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.015780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42851 (local address 127.0.0.1:46740)
0923 20:56:53.031227 (+ 15447us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:53.031257 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:53.031421 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:53.034846 (+ 3425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:53.034851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:53.034879 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:53.035075 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.035089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.037295 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.037302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.039378 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.039398 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.041355 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.041362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.041589 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.041624 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:56:53.041701 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:56:53.041889 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15278,"thread_start_us":16408,"threads_started":1}
I20260923 20:56:53.042374 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.016684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48178)
0923 20:56:53.031896 (+ 15212us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:53.031928 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:53.032083 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:53.035129 (+ 3046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:53.035134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:53.035160 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:53.035576 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.035589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.037616 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.037620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.039604 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.039618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.041351 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.041358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.041579 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.041625 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 20:56:53.041759 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:56:53.041939 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15049,"thread_start_us":16831,"threads_started":1}
I20260923 20:56:53.042348 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.017238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.0.97.1:39007)
0923 20:56:53.034224 (+ 16986us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.034235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.034268 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:56:53.034374 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.034382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.034675 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.034862 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:53.035781 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.037412 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.039814 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.041089 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.041282 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.041543 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.041835 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.042018 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15956,"thread_start_us":15495,"threads_started":1}
I20260923 20:56:53.043498 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.016399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.0.97.3:42851)
0923 20:56:53.034330 (+ 17931us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.034336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.034366 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:56:53.034443 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.034448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.034625 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.034855 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:53.035353 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.037059 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.039637 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.041139 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.042650 (+ 1511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.042905 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.042984 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.043157 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17734,"thread_start_us":17551,"threads_started":1}
I20260923 20:56:53.047160 14696 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbd42a751a940249080cb1f204e6c5c (DEFAULT_TABLE table=test-table [id=63c4b6e20b604f86951b47042a7ad259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:53.048967 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbd42a751a940249080cb1f204e6c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:53.049576 14428 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbd42a751a940249080cb1f204e6c5c (DEFAULT_TABLE table=test-table [id=63c4b6e20b604f86951b47042a7ad259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:53.050843 14428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbd42a751a940249080cb1f204e6c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:53.058095 14780 tablet_bootstrap.cc:492] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Bootstrap starting.
I20260923 20:56:53.062307 14780 tablet_bootstrap.cc:654] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:53.063823 14781 tablet_bootstrap.cc:492] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Bootstrap starting.
I20260923 20:56:53.063797 14780 log.cc:824] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:53.065858 14782 tablet_bootstrap.cc:492] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Bootstrap starting.
I20260923 20:56:53.067854 14781 tablet_bootstrap.cc:654] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:53.069026 14780 tablet_bootstrap.cc:492] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: No bootstrap required, opened a new log
I20260923 20:56:53.069357 14780 ts_tablet_manager.cc:1397] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 20:56:53.069360 14781 log.cc:824] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:53.070062 14782 tablet_bootstrap.cc:654] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:53.071477 14782 log.cc:824] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:53.072991 14781 tablet_bootstrap.cc:492] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: No bootstrap required, opened a new log
I20260923 20:56:53.073323 14781 ts_tablet_manager.cc:1397] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260923 20:56:53.074733 14782 tablet_bootstrap.cc:492] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: No bootstrap required, opened a new log
I20260923 20:56:53.075034 14782 ts_tablet_manager.cc:1397] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260923 20:56:53.080646 14780 raft_consensus.cc:348] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.081279 14780 raft_consensus.cc:374] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:53.081411 14780 raft_consensus.cc:729] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696c6aefbb924d28908dfae3dda3e317, State: Initialized, Role: FOLLOWER
I20260923 20:56:53.082197 14780 consensus_queue.cc:260] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.083544 14627 heartbeater.cc:503] Master 127.0.97.62:42989 was elected leader, sending a full tablet report...
I20260923 20:56:53.083627 14782 raft_consensus.cc:348] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.084100 14782 raft_consensus.cc:374] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:53.084192 14782 raft_consensus.cc:729] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43361c9bb7c408b93c7923859aab40f, State: Initialized, Role: FOLLOWER
I20260923 20:56:53.084282 14780 ts_tablet_manager.cc:1428] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260923 20:56:53.084539 14781 raft_consensus.cc:348] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.084931 14781 raft_consensus.cc:374] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:53.084838 14782 consensus_queue.cc:260] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.085064 14781 raft_consensus.cc:729] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c15a1e7d134833a6002401f2d4fbbb, State: Initialized, Role: FOLLOWER
I20260923 20:56:53.085788 14781 consensus_queue.cc:260] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.086993 14761 heartbeater.cc:503] Master 127.0.97.62:42989 was elected leader, sending a full tablet report...
I20260923 20:56:53.088944 14781 ts_tablet_manager.cc:1428] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260923 20:56:53.090925 14782 ts_tablet_manager.cc:1428] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260923 20:56:53.110628 14494 tablet.cc:2369] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:56:53.144647 14628 tablet.cc:2369] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:56:53.184145 14762 tablet.cc:2369] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:56:53.356316 14787 raft_consensus.cc:482] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:56:53.356693 14787 raft_consensus.cc:504] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.358546 14787 leader_election.cc:288] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
I20260923 20:56:53.368413 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.359024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60695 (local address 127.0.97.3:42851)
0923 20:56:53.359229 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.359235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.360171 (+ 936us) server_negotiation.cc:407] Connection header received
0923 20:56:53.360367 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.360372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.360454 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.360562 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:53.361381 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.362555 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.365846 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.367183 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.367336 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.367450 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.367992 (+ 542us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.368133 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:56:53.368413 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.359110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35233 (local address 127.0.97.2:45919)
0923 20:56:53.359311 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.359319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.360121 (+ 802us) server_negotiation.cc:407] Connection header received
0923 20:56:53.360366 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.360372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.360454 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.360562 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:53.361416 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.362555 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.365846 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.367176 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.367336 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.367451 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.367992 (+ 541us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.368133 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:56:53.368691 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.358947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45919 (local address 127.0.97.1:35233)
0923 20:56:53.359978 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:53.359999 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:53.360181 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:53.360603 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:53.360609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:53.360651 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:53.361262 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.361275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.362709 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.362715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.365604 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.365625 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.367522 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.367530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.367749 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.367828 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 20:56:53.367936 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:56:53.368127 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":307,"spinlock_wait_cycles":3328,"thread_start_us":469,"threads_started":1}
I20260923 20:56:53.368713 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.358875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42851 (local address 127.0.97.1:60695)
0923 20:56:53.359977 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:53.360005 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:53.360222 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:53.360603 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:53.360609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:53.360643 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:53.361168 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.361188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.362709 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.362715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.365604 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:53.365625 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.367534 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.367539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:53.367743 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.367825 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 20:56:53.367936 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 20:56:53.368127 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":287,"spinlock_wait_cycles":40960,"thread_start_us":632,"threads_started":1}
I20260923 20:56:53.370052 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbd42a751a940249080cb1f204e6c5c" candidate_uuid: "a43361c9bb7c408b93c7923859aab40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c15a1e7d134833a6002401f2d4fbbb" is_pre_election: true
I20260923 20:56:53.370052 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbd42a751a940249080cb1f204e6c5c" candidate_uuid: "a43361c9bb7c408b93c7923859aab40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696c6aefbb924d28908dfae3dda3e317" is_pre_election: true
I20260923 20:56:53.370649 14716 raft_consensus.cc:2454] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43361c9bb7c408b93c7923859aab40f in term 0.
I20260923 20:56:53.370649 14582 raft_consensus.cc:2454] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a43361c9bb7c408b93c7923859aab40f in term 0.
I20260923 20:56:53.371395 14384 leader_election.cc:302] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317, a43361c9bb7c408b93c7923859aab40f; no voters:
I20260923 20:56:53.371886 14787 raft_consensus.cc:2781] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:56:53.372058 14787 raft_consensus.cc:482] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:56:53.372166 14787 raft_consensus.cc:3037] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:53.374578 14787 raft_consensus.cc:504] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.375260 14787 leader_election.cc:288] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 1 election: Requested vote from peers 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
I20260923 20:56:53.375849 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbd42a751a940249080cb1f204e6c5c" candidate_uuid: "a43361c9bb7c408b93c7923859aab40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c15a1e7d134833a6002401f2d4fbbb"
I20260923 20:56:53.375850 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbd42a751a940249080cb1f204e6c5c" candidate_uuid: "a43361c9bb7c408b93c7923859aab40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696c6aefbb924d28908dfae3dda3e317"
I20260923 20:56:53.376091 14582 raft_consensus.cc:3037] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:53.376099 14716 raft_consensus.cc:3037] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:53.379693 14582 raft_consensus.cc:2454] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43361c9bb7c408b93c7923859aab40f in term 1.
I20260923 20:56:53.379689 14716 raft_consensus.cc:2454] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a43361c9bb7c408b93c7923859aab40f in term 1.
I20260923 20:56:53.380322 14381 leader_election.cc:302] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696c6aefbb924d28908dfae3dda3e317, a43361c9bb7c408b93c7923859aab40f; no voters:
I20260923 20:56:53.380749 14787 raft_consensus.cc:2781] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:56:53.381229 14787 raft_consensus.cc:686] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 1 LEADER]: Becoming Leader. State: Replica: a43361c9bb7c408b93c7923859aab40f, State: Running, Role: LEADER
I20260923 20:56:53.381785 14787 consensus_queue.cc:237] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } }
I20260923 20:56:53.386700 14304 catalog_manager.cc:5495] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f reported cstate change: term changed from 0 to 1, leader changed from <none> to a43361c9bb7c408b93c7923859aab40f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a43361c9bb7c408b93c7923859aab40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } health_report { overall_health: HEALTHY } } }
I20260923 20:56:53.485520 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.479962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.0.97.1:39007)
0923 20:56:53.480172 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.480177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.480195 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:56:53.480290 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.480295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.480378 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.480517 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:56:53.480904 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.481646 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.483150 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.483594 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.483702 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.484228 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:56:53.484761 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:56:53.484879 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.485094 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.485236 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:56:53.494864 14716 raft_consensus.cc:1261] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Refusing update from remote peer a43361c9bb7c408b93c7923859aab40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:53.494849 14582 raft_consensus.cc:1261] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 1 FOLLOWER]: Refusing update from remote peer a43361c9bb7c408b93c7923859aab40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:53.496376 14791 consensus_queue.cc:1035] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:53.496836 14787 consensus_queue.cc:1035] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:53.529033 14303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:56:53.529392 14303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:56:53.533684 14303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Sending DeleteTablet for 3 replicas of tablet 9dbd42a751a940249080cb1f204e6c5c
I20260923 20:56:53.535645 14696 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbd42a751a940249080cb1f204e6c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46740
I20260923 20:56:53.535728 14562 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbd42a751a940249080cb1f204e6c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57546
I20260923 20:56:53.536171 14428 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbd42a751a940249080cb1f204e6c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48178
I20260923 20:56:53.537884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14365
I20260923 20:56:53.562376 14806 tablet_replica.cc:330] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: stopping tablet replica
I20260923 20:56:53.565801 14806 raft_consensus.cc:2229] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:56:53.569883 14806 raft_consensus.cc:2258] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:56:53.574757 14804 tablet_replica.cc:330] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: stopping tablet replica
I20260923 20:56:53.575032 14805 tablet_replica.cc:330] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: stopping tablet replica
I20260923 20:56:53.575404 14804 raft_consensus.cc:2229] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:56:53.575518 14805 raft_consensus.cc:2229] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:56:53.576066 14804 raft_consensus.cc:2258] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:56:53.576136 14805 raft_consensus.cc:2258] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:56:53.576305 14806 ts_tablet_manager.cc:1905] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:56:53.578462 14804 ts_tablet_manager.cc:1905] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:56:53.578462 14805 ts_tablet_manager.cc:1905] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:56:53.586869 14804 ts_tablet_manager.cc:1918] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:56:53.586870 14805 ts_tablet_manager.cc:1918] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:56:53.587083 14805 log.cc:1196] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/9dbd42a751a940249080cb1f204e6c5c
I20260923 20:56:53.587215 14804 log.cc:1196] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/9dbd42a751a940249080cb1f204e6c5c
I20260923 20:56:53.587860 14805 ts_tablet_manager.cc:1939] T 9dbd42a751a940249080cb1f204e6c5c P 696c6aefbb924d28908dfae3dda3e317: Deleting consensus metadata
I20260923 20:56:53.587860 14804 ts_tablet_manager.cc:1939] T 9dbd42a751a940249080cb1f204e6c5c P 45c15a1e7d134833a6002401f2d4fbbb: Deleting consensus metadata
I20260923 20:56:53.590093 14291 catalog_manager.cc:4841] TS 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851): tablet 9dbd42a751a940249080cb1f204e6c5c (table test-table [id=63c4b6e20b604f86951b47042a7ad259]) successfully deleted
I20260923 20:56:53.590104 14288 catalog_manager.cc:4841] TS 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919): tablet 9dbd42a751a940249080cb1f204e6c5c (table test-table [id=63c4b6e20b604f86951b47042a7ad259]) successfully deleted
Sep 23 20:56:53 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:56:53.592483 14806 ts_tablet_manager.cc:1918] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:56:53.592730 14806 log.cc:1196] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/9dbd42a751a940249080cb1f204e6c5c
I20260923 20:56:53.593398 14806 ts_tablet_manager.cc:1939] T 9dbd42a751a940249080cb1f204e6c5c P a43361c9bb7c408b93c7923859aab40f: Deleting consensus metadata
I20260923 20:56:53.595842 14288 catalog_manager.cc:4841] TS a43361c9bb7c408b93c7923859aab40f (127.0.97.1:39007): tablet 9dbd42a751a940249080cb1f204e6c5c (table test-table [id=63c4b6e20b604f86951b47042a7ad259]) successfully deleted
I20260923 20:56:53.600967 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.540081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.0.97.1:39007)
0923 20:56:53.540361 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.540371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.563337 (+ 22966us) server_negotiation.cc:407] Connection header received
0923 20:56:53.564154 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.564164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.564299 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.564421 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:53.570711 (+ 6290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.572848 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.585084 (+ 12236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.585913 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.586052 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.592229 (+ 6177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:53.592253 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:53.592279 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:53.592339 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:53.596997 (+ 4658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:53.598117 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:53.598124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:53.598127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:53.598326 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:53.599030 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:53.599034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:53.599037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:53.599601 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:53.599853 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.600362 (+ 509us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.600581 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 20:56:53.601754 14488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:53.710351 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14365
I20260923 20:56:53.730201 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14499
Sep 23 20:56:53 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:56:53.746119 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.731007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.0.97.2:45919)
0923 20:56:53.731206 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.731211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.731229 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:56:53.731362 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.731366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.731431 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.731517 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:53.732792 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.733518 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.735012 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.735508 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.735611 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.740254 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:53.740271 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:53.740288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:53.740328 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:53.743338 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:53.744102 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:53.744109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:53.744111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:53.744256 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:53.744604 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:53.744607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:53.744609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:53.744991 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:53.745232 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.745484 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.745656 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:56:53.746824 14622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:53.837009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14499
I20260923 20:56:53.854715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14633
Sep 23 20:56:53 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:56:53.871467 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:53.855494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.0.97.3:42851)
0923 20:56:53.855684 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:53.855688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:53.855707 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:56:53.855850 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:53.855854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:53.855927 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:53.856017 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:53.857294 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.858086 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.859525 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:53.860231 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:53.860388 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:53.865492 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:53.865508 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:53.865526 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:53.865566 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:53.868451 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:53.869202 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:53.869214 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:53.869223 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:53.869367 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:53.869780 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:53.869786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:53.869789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:53.870232 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:53.870539 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:53.870863 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 20:56:53.871097 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:56:53.872191 14756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:53.965992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14633
I20260923 20:56:53.983129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14272
I20260923 20:56:53.984324 14334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:54.095048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14272
I20260923 20:56:54.117055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42989
--webserver_interface=127.0.97.62
--webserver_port=36371
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:54.458912 14812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:54.459301 14812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:54.459470 14812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:54.467000 14812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:54.467192 14812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:54.467308 14812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:54.467417 14812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:54.467507 14812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:54.478053 14812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:54.479506 14812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:54.481005 14812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:54.504817 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:54.504761 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:54.504793 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:54.506193 14812 server_base.cc:1034] running on GCE node
I20260923 20:56:54.507151 14812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:54.508908 14812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:54.510185 14812 hybrid_clock.cc:648] HybridClock initialized: now 1790197014510146 us; error 55 us; skew 500 ppm
I20260923 20:56:54.510644 14812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:54 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:54.518250 14812 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:56:54.520457 14812 webserver.cc:460] Webserver started at http://127.0.97.62:36371/ using document root <none> and password file <none>
I20260923 20:56:54.521350 14812 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:54.521516 14812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:54.528848 14812 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:56:54.532384 14828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:54.534133 14812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:56:54.534340 14812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "64a62a97b099eeb3f8b49412024e282c"
server_key_iv: "79a768450390756ebf331fac3e37b09b"
server_key_version: "encryptionkey@0"
I20260923 20:56:54.535920 14812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:54.554438 14812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:54.559077 14812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:54.559362 14812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:54.579696 14812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42989
I20260923 20:56:54.579845 14879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42989 every 8 connection(s)
I20260923 20:56:54.581449 14812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:56:54.590204 14880 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:56:54.591125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14812
I20260923 20:56:54.591946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39007
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42989
--webserver_port=43869
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:56:54.594785 14880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Bootstrap starting.
I20260923 20:56:54.618530 14880 log.cc:824] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:54.633199 14880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:56:54.633890 14880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Bootstrap complete.
I20260923 20:56:54.644011 14880 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:54.644997 14880 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b8a1b3a2a24ff4ab94dc3f9157e6b9, State: Initialized, Role: FOLLOWER
I20260923 20:56:54.645745 14880 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:54.646098 14880 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:56:54.646246 14880 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:56:54.646453 14880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:56:54.650664 14880 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:54.651243 14880 leader_election.cc:302] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b8a1b3a2a24ff4ab94dc3f9157e6b9; no voters:
I20260923 20:56:54.651935 14880 leader_election.cc:288] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:56:54.652084 14884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:56:54.653262 14884 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 LEADER]: Becoming Leader. State: Replica: e4b8a1b3a2a24ff4ab94dc3f9157e6b9, State: Running, Role: LEADER
I20260923 20:56:54.653993 14884 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:54.654435 14880 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:54.656852 14886 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b8a1b3a2a24ff4ab94dc3f9157e6b9. Latest consensus state: current_term: 2 leader_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } } }
I20260923 20:56:54.656894 14885 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } } }
I20260923 20:56:54.657333 14886 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:54.657338 14885 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:54.666927 14890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:56:54.677726 14890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:56:54.679888 14890 catalog_manager.cc:1259] Loaded cluster ID: e0736177fbc749598462f341fc66d2e0
I20260923 20:56:54.680125 14890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:56:54.684520 14890 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 20:56:54.710531 14890 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 20:56:54.723418 14890 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:56:54.725987 14890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Loaded TSK: 0
I20260923 20:56:54.728233 14890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:56:54.729158 14902 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260923 20:56:54.983345 14882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:54.983645 14882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:54.983799 14882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:54.991592 14882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:54.991724 14882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:54.991901 14882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:56:55.002314 14882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43869
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:55.003726 14882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:55.005200 14882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:55.021011 14907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:55.020985 14908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:55.021055 14910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:55.022171 14882 server_base.cc:1034] running on GCE node
I20260923 20:56:55.022879 14882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:55.023948 14882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:55.025274 14882 hybrid_clock.cc:648] HybridClock initialized: now 1790197015025188 us; error 70 us; skew 500 ppm
I20260923 20:56:55.025640 14882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:55 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:55.029251 14882 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:56:55.031106 14882 webserver.cc:460] Webserver started at http://127.0.97.1:43869/ using document root <none> and password file <none>
I20260923 20:56:55.031893 14882 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:55.032068 14882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:55.038365 14882 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260923 20:56:55.041532 14917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:55.043246 14882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:56:55.043437 14882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "a43361c9bb7c408b93c7923859aab40f"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "1a45ab27b0034a1aa8fa41adc2087f53"
server_key_iv: "259e0559881a73aaf35430f876b3b974"
server_key_version: "encryptionkey@0"
I20260923 20:56:55.044536 14882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:55.062582 14882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:55.067368 14882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:55.067693 14882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:55.068871 14882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:55.070531 14882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:55.070627 14882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:55.070789 14882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:55.070894 14882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:55.109143 14882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39007
I20260923 20:56:55.109200 15029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39007 every 8 connection(s)
I20260923 20:56:55.110706 14882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:56:55.117544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 14882
I20260923 20:56:55.118377 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45919
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42989
--webserver_port=38381
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:56:55 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:55.139684 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:55.113858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:53651)
0923 20:56:55.114838 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:55.114887 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:55.116721 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:55.118771 (+ 2050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:55.118779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:55.119322 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:55.119907 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:55.119926 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.122533 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.122537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:55.123829 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:55.123844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.125506 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.125516 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:55.125881 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:55.126828 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:55.126871 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:55.129489 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:55.134725 (+ 5236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:55.134734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:55.134749 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:55.135259 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:55.135890 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:55.135893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:55.135896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:55.136109 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:55.137240 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:55.137247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:55.137582 (+ 335us) client_negotiation.cc:769] Sending connection context
0923 20:56:55.137967 (+ 385us) client_negotiation.cc:240] Negotiation successful
0923 20:56:55.138497 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":419,"threads_started":1}
I20260923 20:56:55.140166 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:55.113559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53651 (local address 127.0.97.62:42989)
0923 20:56:55.114385 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:55.114398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:55.115073 (+ 675us) server_negotiation.cc:407] Connection header received
0923 20:56:55.117453 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:55.117474 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:55.118093 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:55.118575 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:55.120182 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.122322 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.124019 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.125194 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.125440 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:55.129912 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:55.129928 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:55.129954 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:55.130014 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:55.134446 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:55.135452 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:55.135465 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:55.135476 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:55.135738 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:55.136261 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:55.136270 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:55.136276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:55.137074 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:55.137282 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:55.138232 (+ 950us) server_negotiation.cc:299] Negotiation successful
0923 20:56:55.138784 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":370,"threads_started":1}
I20260923 20:56:55.143414 15030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42989
I20260923 20:56:55.143824 15030 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:55.144923 15030 heartbeater.cc:511] Master 127.0.97.62:42989 requested a full tablet report, sending...
I20260923 20:56:55.147670 14845 ts_manager.cc:194] Registered new tserver with Master: a43361c9bb7c408b93c7923859aab40f (127.0.97.1:39007)
I20260923 20:56:55.149490 14845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53651
W20260923 20:56:55.475489 15034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:55.475814 15034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:55.475912 15034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:55.483477 15034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:55.483610 15034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:55.483728 15034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:56:55.494266 15034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38381
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:55.495600 15034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:55.497005 15034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:55.511641 15039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:55.511675 15040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:55.511724 15042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:55.512681 15034 server_base.cc:1034] running on GCE node
I20260923 20:56:55.513419 15034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:55.514569 15034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:55.515827 15034 hybrid_clock.cc:648] HybridClock initialized: now 1790197015515760 us; error 67 us; skew 500 ppm
I20260923 20:56:55.516186 15034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:55 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:55.519539 15034 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:56:55.521164 15034 webserver.cc:460] Webserver started at http://127.0.97.2:38381/ using document root <none> and password file <none>
I20260923 20:56:55.521927 15034 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:55.522055 15034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:55.528110 15034 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:56:55.531138 15049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:55.532811 15034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:56:55.533042 15034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "696c6aefbb924d28908dfae3dda3e317"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "6f8f8f76941fb5dfa4b72f08720525ef"
server_key_iv: "9417ebec7c753d25be1f286ce57f8a4d"
server_key_version: "encryptionkey@0"
I20260923 20:56:55.534215 15034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:55.561054 15034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:55.565562 15034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:55.565979 15034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:55.567096 15034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:55.568636 15034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:55.568744 15034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:55.568900 15034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:55.569006 15034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:55.605533 15034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45919
I20260923 20:56:55.605641 15161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45919 every 8 connection(s)
I20260923 20:56:55.607105 15034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:56:55.609568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 15034
I20260923 20:56:55.610383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42851
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42989
--webserver_port=44133
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:56:55 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:55.632598 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:55.609964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47235 (local address 127.0.97.62:42989)
0923 20:56:55.610156 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:55.610161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:55.611665 (+ 1504us) server_negotiation.cc:407] Connection header received
0923 20:56:55.614094 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:55.614104 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:55.614240 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:55.614369 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:55.616071 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.617277 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.620104 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.620570 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.620684 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:55.625791 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:55.625796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:55.625799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:55.625899 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:55.628716 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:55.629928 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:55.629932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:55.629934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:55.630074 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:55.630657 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:55.630660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:55.630662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:55.630939 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:55.631054 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:55.632145 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0923 20:56:55.632351 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:56:55.634465 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:55.610625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:47235)
0923 20:56:55.611483 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:55.611524 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:55.613639 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:55.614605 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:55.614614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:55.615213 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:55.615858 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:55.615878 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.617507 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.617514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:55.619902 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:55.619925 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:55.620843 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:55.620853 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:55.621226 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:55.622139 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:55.622176 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:55.625465 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:55.628916 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:55.628930 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:55.628955 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:55.629737 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:55.630196 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:55.630204 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:55.630211 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:55.630526 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:55.631087 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:55.631098 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:55.631635 (+ 537us) client_negotiation.cc:769] Sending connection context
0923 20:56:55.632053 (+ 418us) client_negotiation.cc:240] Negotiation successful
0923 20:56:55.632751 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":352,"threads_started":1}
I20260923 20:56:55.636217 15162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42989
I20260923 20:56:55.636632 15162 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:55.637594 15162 heartbeater.cc:511] Master 127.0.97.62:42989 requested a full tablet report, sending...
I20260923 20:56:55.639441 14845 ts_manager.cc:194] Registered new tserver with Master: 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919)
I20260923 20:56:55.640614 14845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47235
W20260923 20:56:55.964051 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:55.964376 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:55.964547 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:55.972245 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:55.972564 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:55.972793 15165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:56:55.983247 15165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44133
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:55.984773 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:55.986339 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:56.001083 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:56.001125 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:56.001533 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:56.007355 15165 server_base.cc:1034] running on GCE node
I20260923 20:56:56.008194 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:56.009305 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:56.010555 15165 hybrid_clock.cc:648] HybridClock initialized: now 1790197016010505 us; error 37 us; skew 500 ppm
I20260923 20:56:56.010960 15165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:56 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:56.014575 15165 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:56:56.016562 15165 webserver.cc:460] Webserver started at http://127.0.97.3:44133/ using document root <none> and password file <none>
I20260923 20:56:56.017390 15165 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:56.017567 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:56.023542 15165 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:56:56.026672 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:56.028193 15165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:56:56.028417 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "45c15a1e7d134833a6002401f2d4fbbb"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "fdc1e68d74cbe97cd63e8426ad29db3e"
server_key_iv: "e82ae5b2fc75b7d8bb40b0a8f8e4c292"
server_key_version: "encryptionkey@0"
I20260923 20:56:56.029515 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:56.043390 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:56.047938 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:56.048259 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:56.049531 15165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:56.050967 15165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:56.051059 15165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:56.051141 15165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:56.051198 15165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:56.090006 15165 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42851
I20260923 20:56:56.090029 15292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42851 every 8 connection(s)
I20260923 20:56:56.091728 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:56:56.098711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 15165
Sep 23 20:56:56 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:56:56.117712 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.094827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41809 (local address 127.0.97.62:42989)
0923 20:56:56.095086 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.095094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.096389 (+ 1295us) server_negotiation.cc:407] Connection header received
0923 20:56:56.098452 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.098458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.098567 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.098743 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:56.100993 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.102168 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.103887 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.104373 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.104573 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.109924 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:56.109934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:56.109941 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:56.109986 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:56.113639 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.114562 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.114570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.114577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.114706 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.115252 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.115257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.115271 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.115670 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:56.115830 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.116902 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.117159 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":95}
I20260923 20:56:56.118436 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.099804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.0.97.62:42989)
0923 20:56:56.101951 (+ 2147us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.101955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.101971 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:56:56.102015 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.102019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.102090 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.102209 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:56.103808 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.104657 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.106159 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.106842 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.106993 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.111063 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:56.111070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:56.111078 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:56.111137 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:56.115026 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.115964 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.115969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.115972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.116123 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.116561 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.116567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.116570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.116971 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:56.117244 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.117808 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.118091 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1983,"thread_start_us":1787,"threads_started":1}
I20260923 20:56:56.118937 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.095352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:41809)
0923 20:56:56.096178 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.096228 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:56.098113 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:56.098985 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:56.098994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:56.099734 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:56.100729 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.100762 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.102351 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.102375 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.103712 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.103728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.104683 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.104691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.104962 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.105746 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:56.105771 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:56.109534 (+ 3763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:56.113870 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:56.113880 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:56.113900 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:56.114396 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:56.114836 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:56.114841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:56.114845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:56.115064 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:56.115828 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:56.115835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:56.116385 (+ 550us) client_negotiation.cc:769] Sending connection context
0923 20:56:56.116749 (+ 364us) client_negotiation.cc:240] Negotiation successful
0923 20:56:56.117384 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":352,"threads_started":1}
I20260923 20:56:56.120489 15293 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42989
I20260923 20:56:56.120893 15293 heartbeater.cc:464] Registering TS with master...
I20260923 20:56:56.121973 15293 heartbeater.cc:511] Master 127.0.97.62:42989 requested a full tablet report, sending...
I20260923 20:56:56.123654 14845 ts_manager.cc:194] Registered new tserver with Master: 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
I20260923 20:56:56.124491 14845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41809
I20260923 20:56:56.134852 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:56:56.142958 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.135978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.0.97.62:42989)
0923 20:56:56.136169 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.136174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.136572 (+ 398us) server_negotiation.cc:407] Connection header received
0923 20:56:56.136763 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.136768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.136982 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.137194 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:56:56.137810 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.138610 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.140424 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.140907 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.141016 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.141700 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:56:56.142376 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:56:56.142503 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.142663 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.142744 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:56:56.144048 14845 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:56:56.151868 14845 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:56:56.153450 15030 heartbeater.cc:503] Master 127.0.97.62:42989 was elected leader, sending a full tablet report...
I20260923 20:56:56.194635 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.166135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42851 (local address 127.0.0.1:46754)
0923 20:56:56.181441 (+ 15306us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.181471 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:56.181666 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:56.186363 (+ 4697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:56.186375 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:56.186428 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:56.187007 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.187023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.189094 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.189101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.191502 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.191522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.193456 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.193464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.193699 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.193755 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 20:56:56.193940 (+ 185us) client_negotiation.cc:240] Negotiation successful
0923 20:56:56.194155 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15131,"mutex_wait_us":4,"thread_start_us":21446,"threads_started":1}
I20260923 20:56:56.195766 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.166535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.0.97.3:42851)
0923 20:56:56.183008 (+ 16473us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.183024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.183082 (+ 58us) server_negotiation.cc:407] Connection header received
0923 20:56:56.185493 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.185509 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.185955 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.186297 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:56.187304 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.188868 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.191711 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.193169 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.193354 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.193615 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.195193 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.195392 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16161,"thread_start_us":18487,"threads_started":1}
I20260923 20:56:56.195863 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.167960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45919 (local address 127.0.0.1:57566)
0923 20:56:56.187329 (+ 19369us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.187360 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:56.187552 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:56.188412 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:56.188417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:56.188448 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:56.188870 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.188886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.190963 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.190969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.192713 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.192727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.195017 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.195024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.195168 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.195193 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:56:56.195307 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 20:56:56.195448 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19167,"mutex_wait_us":24,"thread_start_us":13136,"threads_started":1}
I20260923 20:56:56.196146 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.168470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.0.97.2:45919)
0923 20:56:56.187272 (+ 18802us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.187282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.187522 (+ 240us) server_negotiation.cc:407] Connection header received
0923 20:56:56.187904 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.187913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.188179 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.188397 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:56.189119 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.190803 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.192919 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.194491 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.194636 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.194849 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.195357 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.195532 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18602,"thread_start_us":18797,"threads_started":1}
I20260923 20:56:56.197937 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.165722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.0.97.1:39007)
0923 20:56:56.188267 (+ 22545us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.188274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.188302 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:56:56.188429 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.188437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.188671 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.188898 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:56.189781 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.192338 (+ 2557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.194769 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.196376 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.196554 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.196783 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.197362 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.197556 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22382,"thread_start_us":22188,"threads_started":1}
I20260923 20:56:56.198570 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.165414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48202)
0923 20:56:56.167393 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.167464 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:56.167704 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:56.188908 (+ 21204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:56.188920 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:56.188955 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:56.189451 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.189463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.192503 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.192510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.194537 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.194560 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.196638 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.196646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.196818 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.196851 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:56:56.196946 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 20:56:56.197105 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1633,"mutex_wait_us":2,"thread_start_us":1447,"threads_started":1}
I20260923 20:56:56.200124 15097 tablet_service.cc:1467] Processing CreateTablet for tablet d6184f37ce8d494085375c9b12d15315 (DEFAULT_TABLE table=test-table [id=e7a953fd3cb94257a9d4b88ee4ba8afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:56.200124 15228 tablet_service.cc:1467] Processing CreateTablet for tablet d6184f37ce8d494085375c9b12d15315 (DEFAULT_TABLE table=test-table [id=e7a953fd3cb94257a9d4b88ee4ba8afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:56.201683 15228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6184f37ce8d494085375c9b12d15315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:56.201684 15097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6184f37ce8d494085375c9b12d15315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:56.202050 14965 tablet_service.cc:1467] Processing CreateTablet for tablet d6184f37ce8d494085375c9b12d15315 (DEFAULT_TABLE table=test-table [id=e7a953fd3cb94257a9d4b88ee4ba8afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:56:56.203290 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6184f37ce8d494085375c9b12d15315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:56:56.216557 15309 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Bootstrap starting.
I20260923 20:56:56.221143 15309 tablet_bootstrap.cc:654] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:56.222224 15310 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Bootstrap starting.
I20260923 20:56:56.222746 15309 log.cc:824] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:56.223166 15311 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Bootstrap starting.
I20260923 20:56:56.226450 15309 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: No bootstrap required, opened a new log
I20260923 20:56:56.226781 15309 ts_tablet_manager.cc:1397] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:56:56.227231 15311 tablet_bootstrap.cc:654] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:56.227674 15310 tablet_bootstrap.cc:654] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Neither blocks nor log segments found. Creating new log.
I20260923 20:56:56.228654 15311 log.cc:824] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:56.228914 15310 log.cc:824] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:56.232402 15311 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: No bootstrap required, opened a new log
I20260923 20:56:56.232748 15311 ts_tablet_manager.cc:1397] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:56:56.232801 15310 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: No bootstrap required, opened a new log
I20260923 20:56:56.233067 15310 ts_tablet_manager.cc:1397] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 20:56:56.235948 15309 raft_consensus.cc:348] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.236433 15309 raft_consensus.cc:374] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:56.236583 15309 raft_consensus.cc:729] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43361c9bb7c408b93c7923859aab40f, State: Initialized, Role: FOLLOWER
I20260923 20:56:56.237296 15309 consensus_queue.cc:260] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.241039 15309 ts_tablet_manager.cc:1428] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260923 20:56:56.244457 15310 raft_consensus.cc:348] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.244457 15311 raft_consensus.cc:348] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.244971 15310 raft_consensus.cc:374] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:56.244971 15311 raft_consensus.cc:374] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:56:56.245095 15310 raft_consensus.cc:729] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c15a1e7d134833a6002401f2d4fbbb, State: Initialized, Role: FOLLOWER
I20260923 20:56:56.245095 15311 raft_consensus.cc:729] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696c6aefbb924d28908dfae3dda3e317, State: Initialized, Role: FOLLOWER
I20260923 20:56:56.245772 15311 consensus_queue.cc:260] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.245772 15310 consensus_queue.cc:260] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.247778 15293 heartbeater.cc:503] Master 127.0.97.62:42989 was elected leader, sending a full tablet report...
I20260923 20:56:56.247915 15311 ts_tablet_manager.cc:1428] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260923 20:56:56.248011 15310 ts_tablet_manager.cc:1428] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 20:56:56.248445 15162 heartbeater.cc:503] Master 127.0.97.62:42989 was elected leader, sending a full tablet report...
I20260923 20:56:56.284413 15317 raft_consensus.cc:482] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:56:56.284691 15317 raft_consensus.cc:504] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.286626 15317 leader_election.cc:288] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a43361c9bb7c408b93c7923859aab40f (127.0.97.1:39007), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
I20260923 20:56:56.294553 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.287173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53717 (local address 127.0.97.3:42851)
0923 20:56:56.287425 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.287434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.288184 (+ 750us) server_negotiation.cc:407] Connection header received
0923 20:56:56.288438 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.288442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.288506 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.288612 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:56.289319 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.290151 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.292621 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.293451 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.293613 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.293735 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.294139 (+ 404us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.294273 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:56:56.294576 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.287047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42619 (local address 127.0.97.1:39007)
0923 20:56:56.287333 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.287343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.288264 (+ 921us) server_negotiation.cc:407] Connection header received
0923 20:56:56.288487 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.288491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.288553 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.288656 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:56:56.289344 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.290151 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.292611 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.293407 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.293570 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.293696 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.294161 (+ 465us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.294307 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 20:56:56.294767 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.286967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.97.2:42619)
0923 20:56:56.288129 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.288144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:56.288325 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:56.288696 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:56.288703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:56.288732 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:56.289173 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.289184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.290306 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.290316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.292433 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.292453 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.293683 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.293689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.293989 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.294016 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 20:56:56.294132 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:56:56.294269 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":180,"thread_start_us":634,"threads_started":1}
I20260923 20:56:56.294900 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.286961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42851 (local address 127.0.97.2:53717)
0923 20:56:56.288061 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.288078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:56.288279 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:56.288629 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:56.288633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:56.288657 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:56:56.289156 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.289167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.290289 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.290295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.292457 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:56.292470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.293695 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.293701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:56.293963 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.294000 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:56:56.294109 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:56:56.294269 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":126,"spinlock_wait_cycles":26368,"thread_start_us":619,"threads_started":1}
I20260923 20:56:56.296100 15248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6184f37ce8d494085375c9b12d15315" candidate_uuid: "696c6aefbb924d28908dfae3dda3e317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c15a1e7d134833a6002401f2d4fbbb" is_pre_election: true
I20260923 20:56:56.296100 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6184f37ce8d494085375c9b12d15315" candidate_uuid: "696c6aefbb924d28908dfae3dda3e317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43361c9bb7c408b93c7923859aab40f" is_pre_election: true
I20260923 20:56:56.296720 15248 raft_consensus.cc:2454] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696c6aefbb924d28908dfae3dda3e317 in term 0.
I20260923 20:56:56.296708 14985 raft_consensus.cc:2454] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696c6aefbb924d28908dfae3dda3e317 in term 0.
I20260923 20:56:56.297504 15053 leader_election.cc:302] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317, a43361c9bb7c408b93c7923859aab40f; no voters:
I20260923 20:56:56.298046 15317 raft_consensus.cc:2781] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:56:56.298204 15317 raft_consensus.cc:482] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:56:56.298333 15317 raft_consensus.cc:3037] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:56.300632 15317 raft_consensus.cc:504] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.301294 15317 leader_election.cc:288] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [CANDIDATE]: Term 1 election: Requested vote from peers a43361c9bb7c408b93c7923859aab40f (127.0.97.1:39007), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
I20260923 20:56:56.301894 14985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6184f37ce8d494085375c9b12d15315" candidate_uuid: "696c6aefbb924d28908dfae3dda3e317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43361c9bb7c408b93c7923859aab40f"
I20260923 20:56:56.301885 15248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6184f37ce8d494085375c9b12d15315" candidate_uuid: "696c6aefbb924d28908dfae3dda3e317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c15a1e7d134833a6002401f2d4fbbb"
I20260923 20:56:56.302129 14985 raft_consensus.cc:3037] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:56.302141 15248 raft_consensus.cc:3037] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:56:56.304440 15248 raft_consensus.cc:2454] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696c6aefbb924d28908dfae3dda3e317 in term 1.
I20260923 20:56:56.304476 14985 raft_consensus.cc:2454] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696c6aefbb924d28908dfae3dda3e317 in term 1.
I20260923 20:56:56.305078 15050 leader_election.cc:302] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317, a43361c9bb7c408b93c7923859aab40f; no voters:
I20260923 20:56:56.305559 15317 raft_consensus.cc:2781] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:56:56.306071 15317 raft_consensus.cc:686] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 1 LEADER]: Becoming Leader. State: Replica: 696c6aefbb924d28908dfae3dda3e317, State: Running, Role: LEADER
I20260923 20:56:56.306619 15317 consensus_queue.cc:237] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:56.311497 14844 catalog_manager.cc:5495] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "696c6aefbb924d28908dfae3dda3e317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } health_report { overall_health: UNKNOWN } } }
I20260923 20:56:56.334753 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.329201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.97.2:45919)
0923 20:56:56.329423 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.329431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.329455 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:56:56.329565 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.329568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.329647 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.329818 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:56:56.330234 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.330953 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.332432 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.332840 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.332984 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.333529 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:56:56.334132 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:56:56.334307 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.334370 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.334458 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 20:56:56.343585 15294 tablet.cc:2369] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:56:56.344843 15248 raft_consensus.cc:1261] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Refusing update from remote peer 696c6aefbb924d28908dfae3dda3e317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:56.344920 14985 raft_consensus.cc:1261] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Refusing update from remote peer 696c6aefbb924d28908dfae3dda3e317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:56:56.346016 15320 consensus_queue.cc:1035] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:56.346424 15317 consensus_queue.cc:1035] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:56:56.360106 15325 mvcc.cc:205] Tried to move back new op lower bound from 7332646978934775808 to 7332646978796457984. Current Snapshot: MvccSnapshot[applied={T|T < 7332646978934775808}]
I20260923 20:56:56.376688 14844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:56:56.376945 14844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:56:56.380699 14844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Sending DeleteTablet for 3 replicas of tablet d6184f37ce8d494085375c9b12d15315
I20260923 20:56:56.382032 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet d6184f37ce8d494085375c9b12d15315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48202
I20260923 20:56:56.382483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14882
I20260923 20:56:56.382386 15228 tablet_service.cc:1514] Processing DeleteTablet for tablet d6184f37ce8d494085375c9b12d15315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46754
I20260923 20:56:56.382848 15097 tablet_service.cc:1514] Processing DeleteTablet for tablet d6184f37ce8d494085375c9b12d15315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:56:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57566
I20260923 20:56:56.397706 15333 tablet_replica.cc:330] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: stopping tablet replica
I20260923 20:56:56.398331 15333 raft_consensus.cc:2229] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:56:56.398964 15333 raft_consensus.cc:2258] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:56:56.401686 15333 ts_tablet_manager.cc:1905] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:56:56.404023 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.384097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.0.97.1:39007)
0923 20:56:56.384280 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.384286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.384305 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:56:56.384367 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.384374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.384458 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.384564 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:56.386688 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.387896 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.389878 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.390533 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.390685 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.394434 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:56.394455 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:56.394475 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:56.394533 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:56.399191 (+ 4658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.400491 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.400502 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.400506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.400707 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.401187 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.401194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.401197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.401725 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:56.402065 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.403327 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.403598 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:56:56.405068 15025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:56.406636 15334 tablet_replica.cc:330] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: stopping tablet replica
I20260923 20:56:56.407354 15334 raft_consensus.cc:2229] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:56:56.408535 15334 raft_consensus.cc:2258] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:56:56.411051 15333 ts_tablet_manager.cc:1918] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:56:56.411255 15333 log.cc:1196] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/d6184f37ce8d494085375c9b12d15315
I20260923 20:56:56.411269 15334 ts_tablet_manager.cc:1905] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:56:56.411819 15333 ts_tablet_manager.cc:1939] T d6184f37ce8d494085375c9b12d15315 P 45c15a1e7d134833a6002401f2d4fbbb: Deleting consensus metadata
I20260923 20:56:56.413908 14832 catalog_manager.cc:4841] TS 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851): tablet d6184f37ce8d494085375c9b12d15315 (table test-table [id=e7a953fd3cb94257a9d4b88ee4ba8afa]) successfully deleted
I20260923 20:56:56.421438 15334 ts_tablet_manager.cc:1918] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:56:56.421666 15334 log.cc:1196] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/d6184f37ce8d494085375c9b12d15315
I20260923 20:56:56.422341 15334 ts_tablet_manager.cc:1939] T d6184f37ce8d494085375c9b12d15315 P 696c6aefbb924d28908dfae3dda3e317: Deleting consensus metadata
I20260923 20:56:56.424539 14829 catalog_manager.cc:4841] TS 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919): tablet d6184f37ce8d494085375c9b12d15315 (table test-table [id=e7a953fd3cb94257a9d4b88ee4ba8afa]) successfully deleted
I20260923 20:56:56.508639 15332 tablet_replica.cc:330] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: stopping tablet replica
I20260923 20:56:56.508878 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14882
I20260923 20:56:56.532948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 15034
W20260923 20:56:56.533068 14829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39007 (error 108)
I20260923 20:56:56.535827 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.535297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48222)
0923 20:56:56.535508 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.535638 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260923 20:56:56.536145 14829 catalog_manager.cc:4584] TS a43361c9bb7c408b93c7923859aab40f (127.0.97.1:39007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6184f37ce8d494085375c9b12d15315: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
I20260923 20:56:56.550523 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.533938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.0.97.2:45919)
0923 20:56:56.534142 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.534150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.534179 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:56:56.534538 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.534545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.534631 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.534779 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:56.536500 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.537333 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.539368 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.539902 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.540008 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.543769 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:56.543785 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:56.543802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:56.543866 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:56.547166 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.548186 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.548193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.548196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.548363 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.548775 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.548778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.548780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.549134 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:56.549401 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.549769 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.550020 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:56:56.551280 15157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:56.590109 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.589486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48230)
0923 20:56:56.589760 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.589912 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 20:56:56.591367 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.590931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48246)
0923 20:56:56.591102 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.591196 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:56:56.652230 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.651726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48260)
0923 20:56:56.651957 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.652057 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 20:56:56.653299 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.652960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48276)
0923 20:56:56.653104 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.653188 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 20:56:56.668231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 15034
I20260923 20:56:56.688362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 15165
I20260923 20:56:56.703034 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.689193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.0.97.3:42851)
0923 20:56:56.689374 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:56.689379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:56.689398 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:56:56.689685 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:56.689692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:56.689983 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:56.690084 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:56.691453 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.692199 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.693544 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:56.694039 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:56.694143 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:56.697199 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:56.697214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:56.697240 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:56.697286 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:56.700191 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.701009 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.701016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.701018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.701188 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:56.701624 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:56.701628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:56.701630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:56.702064 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:56.702290 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:56.702595 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 20:56:56.702783 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:56:56.703760 15288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:56.734464 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.733916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48284)
0923 20:56:56.734167 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.734277 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 20:56:56.735719 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:56.735270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39007 (local address 127.0.0.1:48298)
0923 20:56:56.735509 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:56.735596 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260923 20:56:56.795430 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 15165
I20260923 20:56:56.812378 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 14812
I20260923 20:56:56.813608 14875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:56:56.933643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 14812
I20260923 20:56:56.956234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42989
--webserver_interface=127.0.97.62
--webserver_port=36371
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:57.323354 15340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:57.323742 15340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:57.323910 15340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:57.331662 15340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:56:57.331839 15340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:57.331943 15340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:57.332049 15340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:56:57.332129 15340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:56:57.342541 15340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:57.344027 15340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:57.345419 15340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:57.358173 15345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:57.358246 15346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:57.358203 15348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:57.359599 15340 server_base.cc:1034] running on GCE node
I20260923 20:56:57.360482 15340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:57.362119 15340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:57.363385 15340 hybrid_clock.cc:648] HybridClock initialized: now 1790197017363341 us; error 60 us; skew 500 ppm
I20260923 20:56:57.363798 15340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:57 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:57.367134 15340 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:56:57.368906 15340 webserver.cc:460] Webserver started at http://127.0.97.62:36371/ using document root <none> and password file <none>
I20260923 20:56:57.369688 15340 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:57.369882 15340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:57.375851 15340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 20:56:57.379010 15355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:57.380586 15340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:56:57.380811 15340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "64a62a97b099eeb3f8b49412024e282c"
server_key_iv: "79a768450390756ebf331fac3e37b09b"
server_key_version: "encryptionkey@0"
I20260923 20:56:57.381916 15340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:57.394112 15340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:57.398206 15340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:57.398499 15340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:57.417593 15340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42989
I20260923 20:56:57.417616 15406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42989 every 8 connection(s)
I20260923 20:56:57.419301 15340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:56:57.421021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 15340
I20260923 20:56:57.422061 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39007
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42989
--webserver_port=43869
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:56:57.429574 15407 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:56:57.433424 15407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Bootstrap starting.
I20260923 20:56:57.455392 15407 log.cc:824] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:57.473626 15407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:56:57.474260 15407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9: Bootstrap complete.
I20260923 20:56:57.483914 15407 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:57.484833 15407 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b8a1b3a2a24ff4ab94dc3f9157e6b9, State: Initialized, Role: FOLLOWER
I20260923 20:56:57.485605 15407 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:57.485917 15407 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:56:57.486068 15407 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:56:57.486282 15407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:56:57.490520 15407 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:57.491107 15407 leader_election.cc:302] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b8a1b3a2a24ff4ab94dc3f9157e6b9; no voters:
I20260923 20:56:57.491869 15407 leader_election.cc:288] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:56:57.492017 15411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:56:57.493324 15411 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [term 3 LEADER]: Becoming Leader. State: Replica: e4b8a1b3a2a24ff4ab94dc3f9157e6b9, State: Running, Role: LEADER
I20260923 20:56:57.493921 15411 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } }
I20260923 20:56:57.494256 15407 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:56:57.496572 15413 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b8a1b3a2a24ff4ab94dc3f9157e6b9. Latest consensus state: current_term: 3 leader_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } } }
I20260923 20:56:57.497006 15413 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:57.497587 15412 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b8a1b3a2a24ff4ab94dc3f9157e6b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42989 } } }
I20260923 20:56:57.497897 15412 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b8a1b3a2a24ff4ab94dc3f9157e6b9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:56:57.502497 15418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:56:57.512408 15418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:56:57.514087 15418 catalog_manager.cc:1259] Loaded cluster ID: e0736177fbc749598462f341fc66d2e0
I20260923 20:56:57.514212 15418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 20:56:57.516403 15418 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f71ac8e09ad google::LogMessage::Fail() at ??:0
@ 0x7f71ac8e388a google::LogMessage::SendToLog() at ??:0
@ 0x7f71ac8e04de google::LogMessage::Flush() at ??:0
@ 0x7f71ac8e1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f71c2313512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f71c2310976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f71c2398610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f71c2398439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f71af52c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f71af52f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f71af52f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f71af50aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f71b27e46db start_thread at ??:0
@ 0x7f71ab77a61f clone at ??:0
W20260923 20:56:57.810375 15409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:57.810819 15409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:57.811039 15409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:57.821774 15409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:57.821975 15409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:57.822100 15409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:56:57.832530 15409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43869
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:57.833935 15409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:57.835304 15409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:57.849934 15449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:57.850109 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:57.850078 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:57.851511 15409 server_base.cc:1034] running on GCE node
I20260923 20:56:57.852173 15409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:57.853165 15409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:57.854527 15409 hybrid_clock.cc:648] HybridClock initialized: now 1790197017854486 us; error 55 us; skew 500 ppm
I20260923 20:56:57.854920 15409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:57 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:57.858333 15409 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:56:57.860177 15409 webserver.cc:460] Webserver started at http://127.0.97.1:43869/ using document root <none> and password file <none>
I20260923 20:56:57.860970 15409 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:57.861150 15409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:57.866950 15409 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:56:57.870100 15458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:57.871564 15409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:56:57.871788 15409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "a43361c9bb7c408b93c7923859aab40f"
format_stamp: "Formatted at 2026-09-23 20:56:51 on dist-test-slave-plcg"
server_key: "1a45ab27b0034a1aa8fa41adc2087f53"
server_key_iv: "259e0559881a73aaf35430f876b3b974"
server_key_version: "encryptionkey@0"
I20260923 20:56:57.872895 15409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:57.895659 15409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:57.900050 15409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:57.900365 15409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:57.901440 15409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:57.903417 15465 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260923 20:56:57.909782 15409 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 20:56:57.910001 15409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260923 20:56:57.910157 15409 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 20:56:57.913529 15409 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 20:56:57.913662 15409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.000s
I20260923 20:56:57.914181 15465 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Bootstrap starting.
I20260923 20:56:57.941715 15465 log.cc:824] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:56:57.957204 15465 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:56:57.957474 15409 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39007
I20260923 20:56:57.957515 15572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39007 every 8 connection(s)
I20260923 20:56:57.958029 15465 tablet_bootstrap.cc:492] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Bootstrap complete.
I20260923 20:56:57.958973 15409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:56:57.959460 15465 ts_tablet_manager.cc:1397] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Time spent bootstrapping tablet: real 0.046s user 0.031s sys 0.012s
I20260923 20:56:57.960295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 15409
I20260923 20:56:57.961024 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45919
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42989
--webserver_port=38381
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:56:57.972384 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:57.963404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:51901)
0923 20:56:57.970585 (+ 7181us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:57.970897 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6777,"thread_start_us":6450,"threads_started":1}
I20260923 20:56:57.971957 15465 raft_consensus.cc:348] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:57.973372 15465 raft_consensus.cc:729] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43361c9bb7c408b93c7923859aab40f, State: Initialized, Role: FOLLOWER
W20260923 20:56:57.973547 15573 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42989 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42989: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
I20260923 20:56:57.974220 15465 consensus_queue.cc:260] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:57.974807 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:57.974276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:52379)
0923 20:56:57.974510 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:57.974608 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:56:57.976321 15465 ts_tablet_manager.cc:1428] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260923 20:56:57.976681 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:57.975414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:40577)
0923 20:56:57.976023 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:57.976264 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":279,"threads_started":1}
W20260923 20:56:57.976989 15573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:56:57.977854 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:57.977392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:35051)
0923 20:56:57.977576 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:57.977665 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 20:56:58.307828 15576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:58.308130 15576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:58.308329 15576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:58.315989 15576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:58.316123 15576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:58.316321 15576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:56:58.326857 15576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38381
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:58.328294 15576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:58.329649 15576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:58.343714 15584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:58.343688 15586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:58.343963 15583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:58.344625 15576 server_base.cc:1034] running on GCE node
I20260923 20:56:58.345357 15576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:58.346549 15576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:58.347820 15576 hybrid_clock.cc:648] HybridClock initialized: now 1790197018347778 us; error 43 us; skew 500 ppm
I20260923 20:56:58.348208 15576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:58 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:58.351706 15576 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:56:58.353407 15576 webserver.cc:460] Webserver started at http://127.0.97.2:38381/ using document root <none> and password file <none>
I20260923 20:56:58.354277 15576 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:58.354463 15576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:58.360494 15576 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 20:56:58.363607 15593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:58.365130 15576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:56:58.365341 15576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "696c6aefbb924d28908dfae3dda3e317"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "6f8f8f76941fb5dfa4b72f08720525ef"
server_key_iv: "9417ebec7c753d25be1f286ce57f8a4d"
server_key_version: "encryptionkey@0"
I20260923 20:56:58.366501 15576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:58.395838 15576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:58.400321 15576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:58.400647 15576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:58.401755 15576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:58.403244 15576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:58.403390 15576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:58.403533 15576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:58.403632 15576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:58.439347 15576 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45919
I20260923 20:56:58.439409 15705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45919 every 8 connection(s)
I20260923 20:56:58.440922 15576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:56:58.446054 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:58.444007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:39873)
0923 20:56:58.444788 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:58.445019 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":299,"threads_started":1}
W20260923 20:56:58.446658 15706 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42989 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42989: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
I20260923 20:56:58.447537 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:58.447205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:34793)
0923 20:56:58.447354 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:58.447425 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 20:56:58.448383 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:58.448068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:57225)
0923 20:56:58.448197 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:58.448274 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:56:58.448644 15706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:56:58.450637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 15576
I20260923 20:56:58.451208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42851
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42989
--webserver_port=44133
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:56:58.789294 15709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:56:58.789600 15709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:56:58.789769 15709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:56:58.797544 15709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:56:58.797673 15709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:56:58.797891 15709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:56:58.808071 15709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44133
--tserver_master_addrs=127.0.97.62:42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:56:58.809463 15709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:56:58.810881 15709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:56:58.826283 15715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:58.826242 15714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:56:58.826327 15717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:56:58.827291 15709 server_base.cc:1034] running on GCE node
I20260923 20:56:58.828028 15709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:56:58.829062 15709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:56:58.830384 15709 hybrid_clock.cc:648] HybridClock initialized: now 1790197018830298 us; error 72 us; skew 500 ppm
I20260923 20:56:58.830780 15709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:56:58 dist-test-slave-plcg krb5kdc[14259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:56:58.834345 15709 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:56:58.836089 15709 webserver.cc:460] Webserver started at http://127.0.97.3:44133/ using document root <none> and password file <none>
I20260923 20:56:58.836860 15709 fs_manager.cc:359] Metadata directory not provided
I20260923 20:56:58.837015 15709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:56:58.843138 15709 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:56:58.846401 15724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:58.848016 15709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:56:58.848246 15709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "45c15a1e7d134833a6002401f2d4fbbb"
format_stamp: "Formatted at 2026-09-23 20:56:52 on dist-test-slave-plcg"
server_key: "fdc1e68d74cbe97cd63e8426ad29db3e"
server_key_iv: "e82ae5b2fc75b7d8bb40b0a8f8e4c292"
server_key_version: "encryptionkey@0"
I20260923 20:56:58.849340 15709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:56:58.870684 15709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:56:58.875689 15709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:56:58.876003 15709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:56:58.877074 15709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:56:58.879104 15709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:56:58.879251 15709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:58.879391 15709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:56:58.879477 15709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:56:58.915091 15709 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42851
I20260923 20:56:58.915169 15836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42851 every 8 connection(s)
I20260923 20:56:58.916652 15709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:56:58.921770 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:58.919815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:34661)
0923 20:56:58.920644 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:58.920852 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":338,"threads_started":1}
W20260923 20:56:58.922662 15837 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42989 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42989: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
I20260923 20:56:58.923558 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:58.923219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:33767)
0923 20:56:58.923368 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:58.923441 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:56:58.924407 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:58.924080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:52733)
0923 20:56:58.924215 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:58.924290 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 20:56:58.924911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 15709
W20260923 20:56:58.925233 15837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:56:58.979548 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:58.978798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:48659)
0923 20:56:58.979251 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:58.979344 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":204,"threads_started":1}
I20260923 20:56:59.389407 15842 raft_consensus.cc:482] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:56:59.389724 15842 raft_consensus.cc:504] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:56:59.391505 15842 leader_election.cc:288] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
Sep 23 20:56:59 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 23 20:56:59 dist-test-slave-plcg krb5kdc[14259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:56:59.416682 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:59.391819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45919 (local address 127.0.97.1:35297)
0923 20:56:59.393388 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:59.393407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:59.393528 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:59.395790 (+ 2262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:59.395794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:59.396691 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:59.397479 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:59.397501 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.399243 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.399246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:59.400789 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:59.400802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.401923 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.401931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:59.402259 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:59.403262 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:59.403285 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:59.406434 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:59.411700 (+ 5266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:59.411716 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:59.411744 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:59.412966 (+ 1222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:59.413640 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:59.413646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:59.413651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:59.414006 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:59.415278 (+ 1272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:59.415286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:59.415663 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 20:56:59.415924 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 20:56:59.416353 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"mutex_wait_us":688,"spinlock_wait_cycles":70528,"thread_start_us":703,"threads_started":1}
I20260923 20:56:59.416862 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:59.392203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35297 (local address 127.0.97.2:45919)
0923 20:56:59.393431 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:59.393447 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:59.393485 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:56:59.395145 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:59.395153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:59.395505 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:59.395825 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:59.397725 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.399008 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.400952 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.401641 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.401892 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:59.406873 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:59.406887 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:59.406915 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:59.406981 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:59.411342 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:59.413171 (+ 1829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:59.413182 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:59.413193 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:59.413468 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:59.414145 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:59.414150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:59.414154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:59.414953 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:59.415152 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:59.416159 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0923 20:56:59.416514 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":1073,"threads_started":1}
I20260923 20:56:59.417943 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:59.391801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42851 (local address 127.0.97.1:48123)
0923 20:56:59.392087 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:59.392132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:56:59.392718 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:56:59.395771 (+ 3053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:56:59.395782 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:56:59.396691 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:56:59.397479 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:59.397501 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.399226 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.399234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:59.400763 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:56:59.400779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.401960 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.401967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:56:59.402260 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:59.403140 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:56:59.403181 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:56:59.406652 (+ 3471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:56:59.411700 (+ 5048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:59.411716 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:59.411744 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:59.412966 (+ 1222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:59.413640 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:56:59.413647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:56:59.413651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:56:59.414006 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:56:59.416915 (+ 2909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:56:59.416920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:56:59.417091 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 20:56:59.417267 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 20:56:59.417563 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":145,"spinlock_wait_cycles":25088}
I20260923 20:56:59.418342 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:59.392134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48123 (local address 127.0.97.3:42851)
0923 20:56:59.392714 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0923 20:56:59.392729 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:56:59.392798 (+ 69us) server_negotiation.cc:407] Connection header received
0923 20:56:59.394798 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:56:59.394808 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:56:59.395241 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:56:59.395606 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:56:59.397733 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.399012 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.400937 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:56:59.401639 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:56:59.401901 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:56:59.406946 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:56:59.406956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:56:59.406967 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:56:59.407021 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:56:59.411342 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:59.413171 (+ 1829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:59.413182 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:59.413193 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:59.413468 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:56:59.414145 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:56:59.414151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:56:59.414154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:56:59.414953 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:56:59.415152 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 20:56:59.417494 (+ 2342us) server_negotiation.cc:299] Negotiation successful
0923 20:56:59.417905 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":295,"threads_started":1}
W20260923 20:56:59.420557 15459 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
W20260923 20:56:59.420970 15462 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
I20260923 20:56:59.421116 15462 leader_election.cc:302] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43361c9bb7c408b93c7923859aab40f; no voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317
I20260923 20:56:59.421564 15842 raft_consensus.cc:2726] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:56:59.450253 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:59.449232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:44961)
0923 20:56:59.449800 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:59.449987 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":316,"threads_started":1}
I20260923 20:56:59.926683 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:59.925950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:46659)
0923 20:56:59.926399 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:59.926517 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260923 20:56:59.981408 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:56:59.980591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:38899)
0923 20:56:59.981069 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 20:56:59.981210 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":266,"threads_started":1}
I20260923 20:57:00.451917 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:00.451178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:53413)
0923 20:57:00.451652 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:00.451750 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":250,"threads_started":1}
I20260923 20:57:00.928632 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:00.927808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:39489)
0923 20:57:00.928322 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:00.928411 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260923 20:57:00.983171 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:00.982424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:52631)
0923 20:57:00.982896 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:00.982983 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":241,"threads_started":1}
I20260923 20:57:01.101266 15852 raft_consensus.cc:482] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:57:01.101547 15852 raft_consensus.cc:504] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:57:01.102360 15852 leader_election.cc:288] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
W20260923 20:57:01.103415 15462 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
W20260923 20:57:01.103615 15459 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
I20260923 20:57:01.103794 15459 leader_election.cc:302] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43361c9bb7c408b93c7923859aab40f; no voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317
I20260923 20:57:01.104094 15852 raft_consensus.cc:2726] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:57:01.453881 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:01.452945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:43959)
0923 20:57:01.453565 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:01.453658 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":299,"threads_started":1}
I20260923 20:57:01.930660 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:01.929699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:35091)
0923 20:57:01.930307 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:01.930430 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":311,"threads_started":1}
I20260923 20:57:01.985260 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:01.984282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:54449)
0923 20:57:01.984951 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:01.985046 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":416,"threads_started":1}
I20260923 20:57:02.455605 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:02.454875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:51293)
0923 20:57:02.455329 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:02.455421 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":219,"threads_started":1}
I20260923 20:57:02.932435 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:02.931679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:53303)
0923 20:57:02.932151 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:02.932242 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260923 20:57:02.987370 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:02.986632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:38167)
0923 20:57:02.987106 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:02.987199 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260923 20:57:03.457213 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:03.456503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:53819)
0923 20:57:03.456973 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:03.457053 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":274,"threads_started":1}
I20260923 20:57:03.701900 15860 raft_consensus.cc:482] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:57:03.702138 15860 raft_consensus.cc:504] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:57:03.702885 15860 leader_election.cc:288] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
W20260923 20:57:03.703823 15459 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
W20260923 20:57:03.704007 15462 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
I20260923 20:57:03.704214 15462 leader_election.cc:302] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43361c9bb7c408b93c7923859aab40f; no voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317
I20260923 20:57:03.704487 15860 raft_consensus.cc:2726] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:57:03.934238 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:03.933433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:58943)
0923 20:57:03.933964 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:03.934058 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":237,"threads_started":1}
I20260923 20:57:03.989149 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:03.988301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:39703)
0923 20:57:03.988829 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:03.988928 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":267,"threads_started":1}
I20260923 20:57:04.458987 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:04.458245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:53089)
0923 20:57:04.458716 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:04.458804 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":247,"threads_started":1}
I20260923 20:57:04.936087 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:04.935250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:50243)
0923 20:57:04.935797 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:04.935892 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":293,"threads_started":1}
I20260923 20:57:04.991199 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:04.990391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:38717)
0923 20:57:04.990918 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:04.991010 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":277,"threads_started":1}
I20260923 20:57:05.461372 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:05.460138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:50599)
0923 20:57:05.461076 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:05.461175 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":564,"threads_started":1}
I20260923 20:57:05.938014 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:05.937078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:50413)
0923 20:57:05.937603 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:05.937702 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":250,"threads_started":1}
I20260923 20:57:05.993083 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:05.992290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:37371)
0923 20:57:05.992808 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:05.992900 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260923 20:57:06.463311 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:06.462566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:59653)
0923 20:57:06.463034 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:06.463127 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260923 20:57:06.939644 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:06.938964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:44427)
0923 20:57:06.939373 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:06.939457 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
I20260923 20:57:06.994854 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:06.994159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:55863)
0923 20:57:06.994592 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:06.994681 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":227,"threads_started":1}
I20260923 20:57:07.465075 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:07.464359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:55929)
0923 20:57:07.464826 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:07.464913 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260923 20:57:07.941761 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:07.940834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:35993)
0923 20:57:07.941493 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:07.941593 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":321,"threads_started":1}
I20260923 20:57:07.996886 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:07.996090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:39321)
0923 20:57:07.996598 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:07.996696 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":281,"threads_started":1}
I20260923 20:57:08.467041 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:08.466255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:52939)
0923 20:57:08.466771 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:08.466857 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":279,"threads_started":1}
I20260923 20:57:08.758859 15876 raft_consensus.cc:482] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:57:08.759107 15876 raft_consensus.cc:504] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:57:08.759799 15876 leader_election.cc:288] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
W20260923 20:57:08.760820 15459 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
W20260923 20:57:08.760963 15462 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
I20260923 20:57:08.761125 15462 leader_election.cc:302] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43361c9bb7c408b93c7923859aab40f; no voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317
I20260923 20:57:08.761402 15876 raft_consensus.cc:2726] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:57:08.943564 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:08.942799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:51293)
0923 20:57:08.943286 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:08.943377 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260923 20:57:08.998808 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:08.997951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:40005)
0923 20:57:08.998476 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:08.998589 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":257,"threads_started":1}
I20260923 20:57:09.468735 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:09.468000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:37211)
0923 20:57:09.468459 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:09.468547 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
I20260923 20:57:09.945485 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:09.944700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:37245)
0923 20:57:09.945214 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:09.945312 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":238,"threads_started":1}
I20260923 20:57:10.000542 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:09.999800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:50937)
0923 20:57:10.000253 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:10.000352 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260923 20:57:10.470580 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:10.469758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:57771)
0923 20:57:10.470324 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:10.470408 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":334,"threads_started":1}
I20260923 20:57:10.608206 15883 raft_consensus.cc:482] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:57:10.608412 15883 raft_consensus.cc:504] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a43361c9bb7c408b93c7923859aab40f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39007 } } peers { permanent_uuid: "696c6aefbb924d28908dfae3dda3e317" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45919 } } peers { permanent_uuid: "45c15a1e7d134833a6002401f2d4fbbb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42851 } }
I20260923 20:57:10.609133 15883 leader_election.cc:288] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919), 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851)
W20260923 20:57:10.610230 15462 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45c15a1e7d134833a6002401f2d4fbbb (127.0.97.3:42851): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
W20260923 20:57:10.610411 15459 leader_election.cc:341] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 696c6aefbb924d28908dfae3dda3e317 (127.0.97.2:45919): Not found: Tablet not found: d6184f37ce8d494085375c9b12d15315
I20260923 20:57:10.610532 15459 leader_election.cc:302] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a43361c9bb7c408b93c7923859aab40f; no voters: 45c15a1e7d134833a6002401f2d4fbbb, 696c6aefbb924d28908dfae3dda3e317
I20260923 20:57:10.610771 15883 raft_consensus.cc:2726] T d6184f37ce8d494085375c9b12d15315 P a43361c9bb7c408b93c7923859aab40f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:57:10.947248 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:10.946497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:37023)
0923 20:57:10.946973 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:10.947058 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260923 20:57:11.002489 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:11.001589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:53731)
0923 20:57:11.002168 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:11.002252 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":331,"threads_started":1}
I20260923 20:57:11.472613 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:11.471708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:42883)
0923 20:57:11.472274 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:11.472378 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":312,"threads_started":1}
I20260923 20:57:11.949235 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:11.948283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:35225)
0923 20:57:11.948939 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:11.949033 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":315,"threads_started":1}
I20260923 20:57:12.004349 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:12.003550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:56089)
0923 20:57:12.004038 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:12.004126 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260923 20:57:12.474478 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:12.473618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:45007)
0923 20:57:12.474162 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:12.474264 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":306,"threads_started":1}
I20260923 20:57:12.951288 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:12.950447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:33035)
0923 20:57:12.950919 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:12.951008 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260923 20:57:13.006498 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:13.005730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.1:53327)
0923 20:57:13.006241 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:13.006327 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":283,"threads_started":1}
I20260923 20:57:13.476363 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:13.475629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.2:53193)
0923 20:57:13.476086 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:13.476191 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260923 20:57:13.932802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 15409
I20260923 20:57:13.953564 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:13.952520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42989 (local address 127.0.97.3:57341)
0923 20:57:13.953175 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:13.953306 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":322,"threads_started":1}
I20260923 20:57:13.958932 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:13.934153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.97.1:39007)
0923 20:57:13.935903 (+ 1750us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:13.935910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:13.935942 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:57:13.936024 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:13.936031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:13.936261 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:13.936430 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:13.937929 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:13.938978 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:13.940760 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:13.941951 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:13.942120 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:13.951773 (+ 9653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:13.951797 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:13.951819 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:13.951869 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:13.955546 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:13.956392 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:13.956400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:13.956402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:13.956579 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:13.957074 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:13.957079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:13.957081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:13.957500 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:13.957761 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:13.958198 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 20:57:13.958426 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1601,"thread_start_us":871,"threads_started":1}
I20260923 20:57:13.960479 15568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:14.053583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 15409
I20260923 20:57:14.072065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 15576
I20260923 20:57:14.085799 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:14.072816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.0.97.2:45919)
0923 20:57:14.073305 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:14.073309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:14.073327 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:57:14.073384 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:14.073387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:14.073448 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:14.073541 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:14.074962 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:14.075782 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:14.077089 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:14.077516 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:14.077618 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:14.080357 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:14.080363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:14.080367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:14.080407 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:14.083372 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:14.084112 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:14.084115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:14.084118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:14.084272 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:14.084618 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:14.084621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:14.084622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:14.084881 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:14.085082 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:14.085317 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 20:57:14.085500 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":291,"threads_started":1}
I20260923 20:57:14.086822 15701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:14.173123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 15576
I20260923 20:57:14.191082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 15709
I20260923 20:57:14.205359 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:14.192005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.0.97.3:42851)
0923 20:57:14.192616 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:14.192620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:14.192640 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:57:14.192696 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:14.192700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:14.192769 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:14.192870 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:14.194377 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:14.195169 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:14.196466 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:14.196909 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:14.197026 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:14.199824 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:14.199830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:14.199833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:14.199872 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:14.202732 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:14.203514 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:14.203518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:14.203520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:14.203667 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:14.203987 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:14.203990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:14.203992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:14.204262 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:14.204484 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:14.204736 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 20:57:14.204918 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":325,"threads_started":1}
I20260923 20:57:14.206391 15832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:14.292450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 15709
2026-09-23T20:57:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24951 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:57:14 dist-test-slave-plcg krb5kdc[15903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:57:14 dist-test-slave-plcg krb5kdc[15903](info): set up 2 sockets
Sep 23 20:57:14 dist-test-slave-plcg krb5kdc[15903](info): commencing operation
krb5kdc: starting...
W20260923 20:57:15.422435 388 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:57:15 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:57:15.673805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39595
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:57:16.025271 15916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:16.025585 15916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:16.025760 15916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:16.033411 15916 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:57:16.033528 15916 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:57:16.033641 15916 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:57:16.033738 15916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:57:16.033859 15916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:16.033972 15916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:16.034044 15916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:57:16.034125 15916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:57:16.044553 15916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:16.046054 15916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:16.047443 15916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:57:16.060226 15921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:16.060251 15924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:16.060560 15916 server_base.cc:1034] running on GCE node
W20260923 20:57:16.060330 15922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:16.062597 15916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:16.064126 15916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:16.065390 15916 hybrid_clock.cc:648] HybridClock initialized: now 1790197036065291 us; error 59 us; skew 500 ppm
I20260923 20:57:16.065965 15916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:16 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:16.069432 15916 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:57:16.071111 15916 webserver.cc:460] Webserver started at http://127.0.97.62:33439/ using document root <none> and password file <none>
I20260923 20:57:16.071918 15916 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:16.072044 15916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:16.072412 15916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:57:16.075712 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "03bdb018b9c942bbac04ce5e0cde81d1"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "868c83ea3f0a3d33a5df7f99d91129d6"
server_key_iv: "4bd81f4f57efd78661f29c8e1eba2e6e"
server_key_version: "encryptionkey@0"
I20260923 20:57:16.076628 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "03bdb018b9c942bbac04ce5e0cde81d1"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "868c83ea3f0a3d33a5df7f99d91129d6"
server_key_iv: "4bd81f4f57efd78661f29c8e1eba2e6e"
server_key_version: "encryptionkey@0"
I20260923 20:57:16.082512 15916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:57:16.086374 15931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:16.087945 15916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:57:16.088191 15916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "03bdb018b9c942bbac04ce5e0cde81d1"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "868c83ea3f0a3d33a5df7f99d91129d6"
server_key_iv: "4bd81f4f57efd78661f29c8e1eba2e6e"
server_key_version: "encryptionkey@0"
I20260923 20:57:16.088441 15916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:16.112973 15916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:16.117496 15916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:16.117851 15916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:16.137569 15916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39595
I20260923 20:57:16.137588 15982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39595 every 8 connection(s)
I20260923 20:57:16.139158 15916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:57:16.139609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 15916
I20260923 20:57:16.139842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:57:16.140727 388 external_mini_cluster.cc:1428] Setting key aca6a9c0152017198ff555b3f33b03fc
I20260923 20:57:16.144217 15983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:57:16 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:57:16.158048 15983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Bootstrap starting.
I20260923 20:57:16.162835 15983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Neither blocks nor log segments found. Creating new log.
I20260923 20:57:16.164172 15983 log.cc:824] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:16.167830 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:16.142924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.0.97.62:39595)
0923 20:57:16.143775 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:16.143793 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:16.143859 (+ 66us) server_negotiation.cc:407] Connection header received
0923 20:57:16.146258 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:16.146288 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:16.146803 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:16.147283 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:16.148651 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:16.150807 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:16.152341 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:16.153425 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:16.153618 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:16.158319 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:16.158331 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:16.158351 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:16.158414 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:16.162581 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:16.163503 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:16.163513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:16.163522 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:16.163750 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:16.164191 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:16.164198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:16.164201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:16.165028 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:16.165267 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:16.165962 (+ 695us) server_negotiation.cc:299] Negotiation successful
0923 20:57:16.166436 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":360,"threads_started":1}
I20260923 20:57:16.167966 15983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: No bootstrap required, opened a new log
I20260923 20:57:16.176635 15983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:16.177047 15983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:57:16.177126 15983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bdb018b9c942bbac04ce5e0cde81d1, State: Initialized, Role: FOLLOWER
I20260923 20:57:16.177695 15983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:16.177982 15983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:57:16.178094 15983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:57:16.178237 15983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:57:16.180706 15983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:16.181288 15983 leader_election.cc:302] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bdb018b9c942bbac04ce5e0cde81d1; no voters:
I20260923 20:57:16.182049 15983 leader_election.cc:288] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:57:16.182189 15988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:57:16.182973 15988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 LEADER]: Becoming Leader. State: Replica: 03bdb018b9c942bbac04ce5e0cde81d1, State: Running, Role: LEADER
I20260923 20:57:16.183604 15988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:16.184329 15983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:57:16.187008 15990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bdb018b9c942bbac04ce5e0cde81d1. Latest consensus state: current_term: 1 leader_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } } }
I20260923 20:57:16.187064 15989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } } }
I20260923 20:57:16.187501 15990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:57:16.187507 15989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:57:16.189378 15995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:57:16.195153 15995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:57:16.205091 15995 catalog_manager.cc:1347] Generated new cluster ID: 40c1a028962040f999c1f76cff68697b
I20260923 20:57:16.205302 15995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:57:16.232402 15995 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:57:16.234005 15995 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:57:16.243945 15995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Generated new TSK 0
I20260923 20:57:16.244891 15995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:57:16.318028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39595
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:57:16.679697 16009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:16.680003 16009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:16.680169 16009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:16.688020 16009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:16.688159 16009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:16.688381 16009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:57:16.699826 16009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:16.701306 16009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:16.702746 16009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:57:16.716765 16014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:16.716825 16017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:16.716728 16015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:16.717945 16009 server_base.cc:1034] running on GCE node
I20260923 20:57:16.718729 16009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:16.719830 16009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:16.721112 16009 hybrid_clock.cc:648] HybridClock initialized: now 1790197036721041 us; error 52 us; skew 500 ppm
I20260923 20:57:16.721496 16009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:16 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:16.725133 16009 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:57:16.726925 16009 webserver.cc:460] Webserver started at http://127.0.97.1:38967/ using document root <none> and password file <none>
I20260923 20:57:16.727686 16009 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:16.727859 16009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:16.728211 16009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:57:16.731504 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "7f2cf3fee36149a79d39c74194c79df3"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "20967d945ef60746426508b80b819835"
server_key_iv: "b4016b078d5f87e97caaa9e8d3b62ba1"
server_key_version: "encryptionkey@0"
I20260923 20:57:16.732422 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7f2cf3fee36149a79d39c74194c79df3"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "20967d945ef60746426508b80b819835"
server_key_iv: "b4016b078d5f87e97caaa9e8d3b62ba1"
server_key_version: "encryptionkey@0"
I20260923 20:57:16.738215 16009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:57:16.742360 16024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:16.743901 16009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:57:16.744112 16009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "7f2cf3fee36149a79d39c74194c79df3"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "20967d945ef60746426508b80b819835"
server_key_iv: "b4016b078d5f87e97caaa9e8d3b62ba1"
server_key_version: "encryptionkey@0"
I20260923 20:57:16.744344 16009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:16.771459 16009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:16.776393 16009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:16.776767 16009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:16.777993 16009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:57:16.779594 16009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:57:16.779678 16009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:16.779819 16009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:57:16.779886 16009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:16.818763 16009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42727
I20260923 20:57:16.818821 16136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42727 every 8 connection(s)
I20260923 20:57:16.820204 16009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:57:16.827623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16009
I20260923 20:57:16.827857 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:57:16.828559 388 external_mini_cluster.cc:1428] Setting key 0abc57be74dc2d6c684f229221abb21f
Sep 23 20:57:16 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:57:16.845050 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:16.823237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51573 (local address 127.0.97.62:39595)
0923 20:57:16.823966 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:16.823972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:16.824927 (+ 955us) server_negotiation.cc:407] Connection header received
0923 20:57:16.827026 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:16.827033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:16.827135 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:16.827267 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:16.829158 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:16.830136 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:16.832137 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:16.833143 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:16.833339 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:16.837750 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:16.837759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:16.837767 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:16.837847 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:16.840954 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:16.842213 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:16.842218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:16.842220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:16.842379 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:16.842999 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:16.843003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:16.843005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:16.843328 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:16.843467 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:16.844537 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0923 20:57:16.844769 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":356,"threads_started":1}
I20260923 20:57:16.846673 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:16.823886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:51573)
0923 20:57:16.824700 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:16.824753 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:16.826694 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:16.827544 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:16.827553 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:16.828268 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:57:16.828957 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:16.828979 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:16.830363 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:16.830370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:16.831848 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:16.831865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:16.833474 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:16.833483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:16.833720 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:16.834586 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:57:16.834612 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:57:16.837418 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:57:16.841179 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:16.841194 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:16.841220 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:16.842034 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:16.842524 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:16.842533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:16.842538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:16.842856 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:16.843498 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:57:16.843511 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:57:16.844034 (+ 523us) client_negotiation.cc:769] Sending connection context
0923 20:57:16.844380 (+ 346us) client_negotiation.cc:240] Negotiation successful
0923 20:57:16.845025 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":329,"threads_started":1}
I20260923 20:57:16.848908 16137 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39595
I20260923 20:57:16.849292 16137 heartbeater.cc:464] Registering TS with master...
I20260923 20:57:16.850294 16137 heartbeater.cc:511] Master 127.0.97.62:39595 requested a full tablet report, sending...
I20260923 20:57:16.853489 15948 ts_manager.cc:194] Registered new tserver with Master: 7f2cf3fee36149a79d39c74194c79df3 (127.0.97.1:42727)
I20260923 20:57:16.856112 15948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51573
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:57:16.894883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39595
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:57:17.244275 16143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:17.244644 16143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:17.244832 16143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:17.252722 16143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:17.252907 16143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:17.253108 16143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:57:17.263381 16143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:17.264786 16143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:17.266160 16143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:57:17.280190 16148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:17.280149 16149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:17.280112 16151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:17.281126 16143 server_base.cc:1034] running on GCE node
I20260923 20:57:17.281936 16143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:17.283051 16143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:17.284303 16143 hybrid_clock.cc:648] HybridClock initialized: now 1790197037284198 us; error 98 us; skew 500 ppm
I20260923 20:57:17.284718 16143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:17 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:17.294973 16143 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:57:17.296795 16143 webserver.cc:460] Webserver started at http://127.0.97.2:46859/ using document root <none> and password file <none>
I20260923 20:57:17.297544 16143 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:17.297662 16143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:17.298020 16143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:57:17.301219 16143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "1e86d58a30b444fda58873334b0df369"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "f65ce0b74178d9f72c1899c2cfd767f2"
server_key_iv: "a5f5a123f1f77d50640f1e98599e20a2"
server_key_version: "encryptionkey@0"
I20260923 20:57:17.302201 16143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1e86d58a30b444fda58873334b0df369"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "f65ce0b74178d9f72c1899c2cfd767f2"
server_key_iv: "a5f5a123f1f77d50640f1e98599e20a2"
server_key_version: "encryptionkey@0"
I20260923 20:57:17.308481 16143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:57:17.312716 16158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:17.314528 16143 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:57:17.314754 16143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "1e86d58a30b444fda58873334b0df369"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "f65ce0b74178d9f72c1899c2cfd767f2"
server_key_iv: "a5f5a123f1f77d50640f1e98599e20a2"
server_key_version: "encryptionkey@0"
I20260923 20:57:17.314967 16143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:17.346087 16143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:17.352979 16143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:17.353502 16143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:17.355052 16143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:57:17.357345 16143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:57:17.357470 16143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:17.357620 16143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:57:17.357697 16143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:17.398855 16143 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37661
I20260923 20:57:17.398909 16270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37661 every 8 connection(s)
I20260923 20:57:17.400257 16143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:57:17.410696 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16143
I20260923 20:57:17.410917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:57:17.411439 388 external_mini_cluster.cc:1428] Setting key dc76ca9d6b52f3dd0632b3e8e5fd4dd8
Sep 23 20:57:17 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:57:17.423863 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:17.403221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34513 (local address 127.0.97.62:39595)
0923 20:57:17.403890 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:17.403899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:17.404881 (+ 982us) server_negotiation.cc:407] Connection header received
0923 20:57:17.406916 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:17.406921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:17.407017 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:17.407148 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:17.408768 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.409518 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.411406 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.411949 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.412101 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:17.416226 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:17.416235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:17.416243 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:17.416306 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:17.420140 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:17.421030 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:17.421038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:17.421045 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:17.421219 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:17.421662 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:17.421669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:17.421675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:17.422123 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:17.422290 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:17.423109 (+ 819us) server_negotiation.cc:299] Negotiation successful
0923 20:57:17.423422 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":316,"threads_started":1}
I20260923 20:57:17.424907 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:17.403896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:34513)
0923 20:57:17.404689 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:17.404742 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:17.406599 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:17.407415 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:17.407424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:17.407994 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:57:17.408575 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:17.408593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.409699 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.409709 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:17.411234 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:17.411248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.412213 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.412222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:17.412469 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:17.413235 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:57:17.413273 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:57:17.415924 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:57:17.420351 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:17.420359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:17.420375 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:17.420867 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:17.421337 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:17.421340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:17.421343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:17.421536 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:17.422275 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:57:17.422289 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:57:17.422642 (+ 353us) client_negotiation.cc:769] Sending connection context
0923 20:57:17.422997 (+ 355us) client_negotiation.cc:240] Negotiation successful
0923 20:57:17.423607 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":328,"threads_started":1}
I20260923 20:57:17.426904 16271 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39595
I20260923 20:57:17.427246 16271 heartbeater.cc:464] Registering TS with master...
I20260923 20:57:17.428021 16271 heartbeater.cc:511] Master 127.0.97.62:39595 requested a full tablet report, sending...
I20260923 20:57:17.429577 15948 ts_manager.cc:194] Registered new tserver with Master: 1e86d58a30b444fda58873334b0df369 (127.0.97.2:37661)
I20260923 20:57:17.430490 15948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34513
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:57:17.470604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39595
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:57:17.832099 16277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:17.832419 16277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:17.832520 16277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:17.840082 16277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:17.840212 16277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:17.840327 16277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:57:17.850515 16277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:17.851843 16277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:17.853104 16277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 20:57:17.860320 16137 heartbeater.cc:503] Master 127.0.97.62:39595 was elected leader, sending a full tablet report...
W20260923 20:57:17.867548 16283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:17.867467 16282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:17.867508 16285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:17.868848 16277 server_base.cc:1034] running on GCE node
I20260923 20:57:17.869568 16277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:17.870784 16277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:17.872080 16277 hybrid_clock.cc:648] HybridClock initialized: now 1790197037872034 us; error 63 us; skew 500 ppm
I20260923 20:57:17.872458 16277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:17 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:17.876175 16277 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:57:17.878089 16277 webserver.cc:460] Webserver started at http://127.0.97.3:39003/ using document root <none> and password file <none>
I20260923 20:57:17.878870 16277 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:17.879047 16277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:17.879415 16277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:57:17.882892 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "d11fdadf3b674b309540271f7b51fa80"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "d2053a1569aaa9a4449ea05d21697b77"
server_key_iv: "76a3a390e157775e41286bc7f3bd7c77"
server_key_version: "encryptionkey@0"
I20260923 20:57:17.883805 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d11fdadf3b674b309540271f7b51fa80"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "d2053a1569aaa9a4449ea05d21697b77"
server_key_iv: "76a3a390e157775e41286bc7f3bd7c77"
server_key_version: "encryptionkey@0"
I20260923 20:57:17.890851 16277 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.007s
I20260923 20:57:17.895875 16292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:17.897457 16277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:57:17.897710 16277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "d11fdadf3b674b309540271f7b51fa80"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "d2053a1569aaa9a4449ea05d21697b77"
server_key_iv: "76a3a390e157775e41286bc7f3bd7c77"
server_key_version: "encryptionkey@0"
I20260923 20:57:17.898012 16277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:17.920204 16277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:17.924990 16277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:17.925326 16277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:17.926481 16277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:57:17.927969 16277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:57:17.928086 16277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:17.928241 16277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:57:17.928314 16277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:17.964599 16277 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41519
I20260923 20:57:17.964663 16404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41519 every 8 connection(s)
I20260923 20:57:17.966044 16277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:57:17.967358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16277
I20260923 20:57:17.967542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:57:17.968122 388 external_mini_cluster.cc:1428] Setting key f82f103f4380838e6eb48a770b43515d
Sep 23 20:57:17 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:57:17.988565 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:17.969306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32885 (local address 127.0.97.62:39595)
0923 20:57:17.969928 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:17.969934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:17.970986 (+ 1052us) server_negotiation.cc:407] Connection header received
0923 20:57:17.972991 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:17.972996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:17.973084 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:17.973190 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:17.974967 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.975777 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.977484 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.978022 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.978235 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:17.982177 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:17.982182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:17.982186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:17.982227 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:17.985099 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:17.986037 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:17.986042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:17.986044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:17.986176 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:17.986643 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:17.986653 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:17.986655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:17.986939 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:17.987119 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:17.987875 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 20:57:17.988132 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":276,"threads_started":1}
I20260923 20:57:17.989638 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:17.969865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:32885)
0923 20:57:17.970811 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:17.970858 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:17.972672 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:17.973436 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:17.973445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:17.974142 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:57:17.974762 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:17.974780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.975953 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.975957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:17.977306 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:17.977322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:17.978300 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:17.978305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:17.978521 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:17.979301 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:57:17.979326 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:57:17.981903 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:57:17.985270 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:17.985280 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:17.985299 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:17.985896 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:17.986301 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:17.986306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:17.986310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:17.986517 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:17.987076 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:57:17.987086 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:57:17.987476 (+ 390us) client_negotiation.cc:769] Sending connection context
0923 20:57:17.987766 (+ 290us) client_negotiation.cc:240] Negotiation successful
0923 20:57:17.988359 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":397,"threads_started":1}
I20260923 20:57:17.991103 16405 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39595
I20260923 20:57:17.991458 16405 heartbeater.cc:464] Registering TS with master...
I20260923 20:57:17.992254 16405 heartbeater.cc:511] Master 127.0.97.62:39595 requested a full tablet report, sending...
I20260923 20:57:17.993778 15948 ts_manager.cc:194] Registered new tserver with Master: d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519)
I20260923 20:57:17.994668 15948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32885
I20260923 20:57:17.996783 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:57:18.016023 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.002046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.0.97.62:39595)
0923 20:57:18.002308 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.002315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.002564 (+ 249us) server_negotiation.cc:407] Connection header received
0923 20:57:18.002799 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.002805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.002881 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.003016 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:18.004781 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.005560 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.007081 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.007510 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.007627 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.010201 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:18.010206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:18.010209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:18.010249 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:18.012910 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.013792 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.013797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.013799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.014018 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.014467 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.014471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.014473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.014848 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:18.015069 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.015450 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.015718 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:57:18.020756 15948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:57:18.027901 15948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:57:18.055256 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.041255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41519 (local address 127.0.0.1:34380)
0923 20:57:18.042905 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:18.042950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:18.043129 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:18.047915 (+ 4786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:18.047922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:18.047951 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:18.048409 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.048426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.050359 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.050365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.052476 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.052489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.054160 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.054168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.054394 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.054461 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 20:57:18.054583 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 20:57:18.054790 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":20,"thread_start_us":4803,"threads_started":1}
I20260923 20:57:18.056087 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.041498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.0.97.3:41519)
0923 20:57:18.047126 (+ 5628us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.047134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.047166 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:57:18.047276 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.047284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.047618 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.047877 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:18.048637 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.050208 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.052647 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.053631 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.055119 (+ 1488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.055421 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.055501 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.055685 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5433,"thread_start_us":5280,"threads_started":1}
I20260923 20:57:18.057209 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.043442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.97.2:37661)
0923 20:57:18.044019 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.044028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.047626 (+ 3598us) server_negotiation.cc:407] Connection header received
0923 20:57:18.047763 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.047771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.048005 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.048215 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:18.049260 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.050973 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.054174 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.055732 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.055838 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.056018 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.056728 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.056892 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":835,"threads_started":1}
I20260923 20:57:18.057227 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.042971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37661 (local address 127.0.0.1:54570)
0923 20:57:18.047474 (+ 4503us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:18.047492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:18.047610 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:18.048514 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:18.048519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:18.048549 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:18.049069 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.049083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.051157 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.051162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.053951 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.053972 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.056251 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.056258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.056470 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.056559 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 20:57:18.056669 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:57:18.056810 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4376,"mutex_wait_us":239,"thread_start_us":4229,"threads_started":1}
I20260923 20:57:18.057256 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.044088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.0.97.1:42727)
0923 20:57:18.044695 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.044706 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.046504 (+ 1798us) server_negotiation.cc:407] Connection header received
0923 20:57:18.047727 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.047736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.048005 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.048191 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:18.049352 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.050973 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.054166 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.055740 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.055838 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.056061 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.056742 (+ 681us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.056913 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":371,"threads_started":1}
I20260923 20:57:18.058037 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.041964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:39132)
0923 20:57:18.046373 (+ 4409us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:18.046398 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:18.047517 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:18.048484 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:18.048491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:18.048523 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:18.049107 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.049124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.051156 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.051162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.053971 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.053980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.056214 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.056225 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.056469 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.056568 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 20:57:18.057456 (+ 888us) client_negotiation.cc:240] Negotiation successful
0923 20:57:18.057606 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4279,"mutex_wait_us":253,"spinlock_wait_cycles":3456,"thread_start_us":428,"threads_started":1}
I20260923 20:57:18.060593 16340 tablet_service.cc:1467] Processing CreateTablet for tablet 726ee8ffde414cd6beaa06ec4ac8961d (DEFAULT_TABLE table=test-table [id=904f42f9d6164789890ab9c25421ff1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:57:18.060707 16206 tablet_service.cc:1467] Processing CreateTablet for tablet 726ee8ffde414cd6beaa06ec4ac8961d (DEFAULT_TABLE table=test-table [id=904f42f9d6164789890ab9c25421ff1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:57:18.062094 16340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726ee8ffde414cd6beaa06ec4ac8961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:57:18.062129 16206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726ee8ffde414cd6beaa06ec4ac8961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:57:18.062793 16072 tablet_service.cc:1467] Processing CreateTablet for tablet 726ee8ffde414cd6beaa06ec4ac8961d (DEFAULT_TABLE table=test-table [id=904f42f9d6164789890ab9c25421ff1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:57:18.064597 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 726ee8ffde414cd6beaa06ec4ac8961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:57:18.074358 16424 tablet_bootstrap.cc:492] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Bootstrap starting.
I20260923 20:57:18.077597 16425 tablet_bootstrap.cc:492] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Bootstrap starting.
I20260923 20:57:18.079226 16424 tablet_bootstrap.cc:654] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Neither blocks nor log segments found. Creating new log.
I20260923 20:57:18.080514 16426 tablet_bootstrap.cc:492] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Bootstrap starting.
I20260923 20:57:18.080650 16424 log.cc:824] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:18.082362 16425 tablet_bootstrap.cc:654] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Neither blocks nor log segments found. Creating new log.
I20260923 20:57:18.083652 16425 log.cc:824] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:18.084048 16426 tablet_bootstrap.cc:654] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Neither blocks nor log segments found. Creating new log.
I20260923 20:57:18.085139 16424 tablet_bootstrap.cc:492] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: No bootstrap required, opened a new log
I20260923 20:57:18.085155 16426 log.cc:824] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:18.085525 16424 ts_tablet_manager.cc:1397] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260923 20:57:18.088886 16426 tablet_bootstrap.cc:492] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: No bootstrap required, opened a new log
I20260923 20:57:18.089257 16426 ts_tablet_manager.cc:1397] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260923 20:57:18.092844 16425 tablet_bootstrap.cc:492] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: No bootstrap required, opened a new log
I20260923 20:57:18.093158 16425 ts_tablet_manager.cc:1397] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260923 20:57:18.097465 16426 raft_consensus.cc:348] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.098052 16426 raft_consensus.cc:374] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:57:18.097766 16424 raft_consensus.cc:348] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.098186 16426 raft_consensus.cc:729] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2cf3fee36149a79d39c74194c79df3, State: Initialized, Role: FOLLOWER
I20260923 20:57:18.098276 16424 raft_consensus.cc:374] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:57:18.098385 16424 raft_consensus.cc:729] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11fdadf3b674b309540271f7b51fa80, State: Initialized, Role: FOLLOWER
I20260923 20:57:18.099031 16426 consensus_queue.cc:260] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.099095 16424 consensus_queue.cc:260] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.106431 16425 raft_consensus.cc:348] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.106994 16425 raft_consensus.cc:374] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:57:18.107085 16426 ts_tablet_manager.cc:1428] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260923 20:57:18.107153 16425 raft_consensus.cc:729] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e86d58a30b444fda58873334b0df369, State: Initialized, Role: FOLLOWER
I20260923 20:57:18.107358 16424 ts_tablet_manager.cc:1428] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260923 20:57:18.107740 16405 heartbeater.cc:503] Master 127.0.97.62:39595 was elected leader, sending a full tablet report...
I20260923 20:57:18.107934 16425 consensus_queue.cc:260] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.109128 16271 heartbeater.cc:503] Master 127.0.97.62:39595 was elected leader, sending a full tablet report...
I20260923 20:57:18.109866 16425 ts_tablet_manager.cc:1428] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
W20260923 20:57:18.152374 16272 tablet.cc:2369] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:57:18.218242 16406 tablet.cc:2369] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:57:18.293030 16432 raft_consensus.cc:482] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:57:18.293370 16432 raft_consensus.cc:504] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.295164 16432 leader_election.cc:288] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519), 7f2cf3fee36149a79d39c74194c79df3 (127.0.97.1:42727)
I20260923 20:57:18.303191 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.295711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50827 (local address 127.0.97.1:42727)
0923 20:57:18.295898 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.295904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.296720 (+ 816us) server_negotiation.cc:407] Connection header received
0923 20:57:18.296916 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.296922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.297005 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.297114 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:18.297991 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.298947 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.301078 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.302068 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.302218 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.302345 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.302774 (+ 429us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.302910 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:57:18.303195 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.295647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49703 (local address 127.0.97.3:41519)
0923 20:57:18.295857 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.295863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.296710 (+ 847us) server_negotiation.cc:407] Connection header received
0923 20:57:18.296916 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.296922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.297005 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.297114 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:18.298026 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.298930 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.301127 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.302110 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.302280 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.302389 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.302816 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.302944 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:57:18.303401 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.295594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.97.2:50827)
0923 20:57:18.296612 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:18.296628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:18.296779 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:18.297150 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:18.297155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:18.297194 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:18.297735 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.297756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.299124 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.299128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.300888 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.300907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.302335 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.302343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.302530 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.302635 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 20:57:18.302723 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 20:57:18.302885 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":106,"spinlock_wait_cycles":48256,"thread_start_us":482,"threads_started":1}
I20260923 20:57:18.303429 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.295539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41519 (local address 127.0.97.2:49703)
0923 20:57:18.296535 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:18.296563 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:18.296749 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:18.297150 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:18.297155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:18.297184 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:18.297777 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.297792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.299072 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.299077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.300922 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:18.300936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.302373 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.302380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:18.302552 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.302651 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 20:57:18.302772 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 20:57:18.302975 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":121,"thread_start_us":542,"threads_started":1}
I20260923 20:57:18.304647 16360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726ee8ffde414cd6beaa06ec4ac8961d" candidate_uuid: "1e86d58a30b444fda58873334b0df369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11fdadf3b674b309540271f7b51fa80" is_pre_election: true
I20260923 20:57:18.304649 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726ee8ffde414cd6beaa06ec4ac8961d" candidate_uuid: "1e86d58a30b444fda58873334b0df369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2cf3fee36149a79d39c74194c79df3" is_pre_election: true
I20260923 20:57:18.305166 16092 raft_consensus.cc:2454] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e86d58a30b444fda58873334b0df369 in term 0.
I20260923 20:57:18.305166 16360 raft_consensus.cc:2454] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e86d58a30b444fda58873334b0df369 in term 0.
I20260923 20:57:18.305945 16162 leader_election.cc:302] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e86d58a30b444fda58873334b0df369, 7f2cf3fee36149a79d39c74194c79df3, d11fdadf3b674b309540271f7b51fa80; no voters:
I20260923 20:57:18.306603 16432 raft_consensus.cc:2781] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:57:18.306738 16432 raft_consensus.cc:482] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:57:18.306814 16432 raft_consensus.cc:3037] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:57:18.309072 16432 raft_consensus.cc:504] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.309760 16432 leader_election.cc:288] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [CANDIDATE]: Term 1 election: Requested vote from peers d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519), 7f2cf3fee36149a79d39c74194c79df3 (127.0.97.1:42727)
I20260923 20:57:18.310330 16360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726ee8ffde414cd6beaa06ec4ac8961d" candidate_uuid: "1e86d58a30b444fda58873334b0df369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11fdadf3b674b309540271f7b51fa80"
I20260923 20:57:18.310357 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "726ee8ffde414cd6beaa06ec4ac8961d" candidate_uuid: "1e86d58a30b444fda58873334b0df369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2cf3fee36149a79d39c74194c79df3"
I20260923 20:57:18.310580 16360 raft_consensus.cc:3037] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:57:18.310586 16092 raft_consensus.cc:3037] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:57:18.313951 16092 raft_consensus.cc:2454] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e86d58a30b444fda58873334b0df369 in term 1.
I20260923 20:57:18.313953 16360 raft_consensus.cc:2454] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e86d58a30b444fda58873334b0df369 in term 1.
I20260923 20:57:18.314558 16160 leader_election.cc:302] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e86d58a30b444fda58873334b0df369, d11fdadf3b674b309540271f7b51fa80; no voters:
I20260923 20:57:18.314944 16432 raft_consensus.cc:2781] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:57:18.315464 16432 raft_consensus.cc:686] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 1 LEADER]: Becoming Leader. State: Replica: 1e86d58a30b444fda58873334b0df369, State: Running, Role: LEADER
I20260923 20:57:18.315979 16432 consensus_queue.cc:237] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } }
I20260923 20:57:18.321018 15948 catalog_manager.cc:5495] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e86d58a30b444fda58873334b0df369 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1e86d58a30b444fda58873334b0df369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } health_report { overall_health: HEALTHY } } }
W20260923 20:57:18.323083 16138 tablet.cc:2369] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:57:18.360138 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.354811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.0.97.2:37661)
0923 20:57:18.354998 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.355002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.355021 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:57:18.355225 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.355229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.355318 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.355455 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:57:18.355828 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.356564 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.358125 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.358535 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.358645 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.359111 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:57:18.359617 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:57:18.359743 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.359849 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.359922 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:57:18.368697 16092 raft_consensus.cc:1261] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 1 FOLLOWER]: Refusing update from remote peer 1e86d58a30b444fda58873334b0df369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:57:18.368695 16360 raft_consensus.cc:1261] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Refusing update from remote peer 1e86d58a30b444fda58873334b0df369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:57:18.369763 16435 consensus_queue.cc:1035] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:57:18.370224 16432 consensus_queue.cc:1035] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:57:18.383833 16439 mvcc.cc:205] Tried to move back new op lower bound from 7332647069147910144 to 7332647068946214912. Current Snapshot: MvccSnapshot[applied={T|T < 7332647069147910144}]
I20260923 20:57:18.396539 15947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:57:18.396924 15947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:57:18.401717 15947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Sending DeleteTablet for 3 replicas of tablet 726ee8ffde414cd6beaa06ec4ac8961d
I20260923 20:57:18.403138 16072 tablet_service.cc:1514] Processing DeleteTablet for tablet 726ee8ffde414cd6beaa06ec4ac8961d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:57:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39132
I20260923 20:57:18.403244 16340 tablet_service.cc:1514] Processing DeleteTablet for tablet 726ee8ffde414cd6beaa06ec4ac8961d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:57:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34380
I20260923 20:57:18.404362 16206 tablet_service.cc:1514] Processing DeleteTablet for tablet 726ee8ffde414cd6beaa06ec4ac8961d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:57:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54570
I20260923 20:57:18.404829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 16009
I20260923 20:57:18.426594 16448 tablet_replica.cc:330] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: stopping tablet replica
I20260923 20:57:18.427412 16448 raft_consensus.cc:2229] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 23 20:57:18 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:57:18.432363 16448 raft_consensus.cc:2258] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:57:18.433656 16450 tablet_replica.cc:330] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: stopping tablet replica
I20260923 20:57:18.434290 16450 raft_consensus.cc:2229] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:57:18.435168 16450 raft_consensus.cc:2258] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:57:18.435431 16448 ts_tablet_manager.cc:1905] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:57:18.443873 16448 ts_tablet_manager.cc:1918] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:57:18.444152 16448 log.cc:1196] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/726ee8ffde414cd6beaa06ec4ac8961d
I20260923 20:57:18.444617 16449 tablet_replica.cc:330] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: stopping tablet replica
I20260923 20:57:18.444772 16448 ts_tablet_manager.cc:1939] T 726ee8ffde414cd6beaa06ec4ac8961d P 7f2cf3fee36149a79d39c74194c79df3: Deleting consensus metadata
I20260923 20:57:18.445226 16449 raft_consensus.cc:2229] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:57:18.445972 16450 ts_tablet_manager.cc:1905] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:57:18.446341 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.406289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.97.1:42727)
0923 20:57:18.406519 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.406528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.411815 (+ 5287us) server_negotiation.cc:407] Connection header received
0923 20:57:18.412806 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.412811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.412918 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.413001 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:18.417271 (+ 4270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.418714 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.424696 (+ 5982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.425401 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.425526 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.436489 (+ 10963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:18.436504 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:18.436522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:18.436599 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:18.440802 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.442673 (+ 1871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.442685 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.442688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.442906 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.443820 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.443827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.443833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.444423 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:18.444784 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.445565 (+ 781us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.445902 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:57:18.447070 15935 catalog_manager.cc:4841] TS 7f2cf3fee36149a79d39c74194c79df3 (127.0.97.1:42727): tablet 726ee8ffde414cd6beaa06ec4ac8961d (table test-table [id=904f42f9d6164789890ab9c25421ff1d]) successfully deleted
I20260923 20:57:18.448081 16132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:18.448815 16449 raft_consensus.cc:2258] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:57:18.453333 16449 ts_tablet_manager.cc:1905] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:57:18.460875 16449 ts_tablet_manager.cc:1918] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:57:18.461163 16449 log.cc:1196] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/726ee8ffde414cd6beaa06ec4ac8961d
I20260923 20:57:18.461715 16449 ts_tablet_manager.cc:1939] T 726ee8ffde414cd6beaa06ec4ac8961d P d11fdadf3b674b309540271f7b51fa80: Deleting consensus metadata
I20260923 20:57:18.463814 15933 catalog_manager.cc:4841] TS d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519): tablet 726ee8ffde414cd6beaa06ec4ac8961d (table test-table [id=904f42f9d6164789890ab9c25421ff1d]) successfully deleted
I20260923 20:57:18.465497 16450 ts_tablet_manager.cc:1918] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:57:18.465710 16450 log.cc:1196] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/726ee8ffde414cd6beaa06ec4ac8961d
I20260923 20:57:18.466209 16450 ts_tablet_manager.cc:1939] T 726ee8ffde414cd6beaa06ec4ac8961d P 1e86d58a30b444fda58873334b0df369: Deleting consensus metadata
I20260923 20:57:18.467792 15932 catalog_manager.cc:4841] TS 1e86d58a30b444fda58873334b0df369 (127.0.97.2:37661): tablet 726ee8ffde414cd6beaa06ec4ac8961d (table test-table [id=904f42f9d6164789890ab9c25421ff1d]) successfully deleted
I20260923 20:57:18.601922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 16009
I20260923 20:57:18.620360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 16143
Sep 23 20:57:18 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:57:18.639644 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.621211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.0.97.2:37661)
0923 20:57:18.621447 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.621454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.621480 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:57:18.621552 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.621558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.621646 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.621793 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:18.623298 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.624761 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.626347 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.627067 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.627211 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.631829 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:18.631851 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:18.631875 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:18.631939 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:18.636407 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.637255 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.637271 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.637276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.637460 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.637885 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.637891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.637895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.638378 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:18.638757 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.638928 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.639200 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:57:18.640535 16266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:18.755575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 16143
I20260923 20:57:18.776379 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 16277
Sep 23 20:57:18 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:57:18.792714 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:18.777184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.0.97.3:41519)
0923 20:57:18.777376 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:18.777382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:18.777402 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:57:18.777609 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:18.777613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:18.777683 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:18.777796 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:18.779098 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.779847 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.781212 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:18.781619 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:18.781759 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:18.786608 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:18.786624 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:18.786642 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:18.786688 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:18.789747 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.790580 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.790588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.790590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.790755 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:18.791088 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:18.791092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:18.791094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:18.791453 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:18.791683 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:18.792096 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 20:57:18.792377 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:57:18.793519 16400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:18.884845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 16277
I20260923 20:57:18.902551 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 15916
I20260923 20:57:18.903741 15978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:19.022083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 15916
I20260923 20:57:39.045020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39595
--webserver_interface=127.0.97.62
--webserver_port=33439
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:57:39.402972 16456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:39.403306 16456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:39.403488 16456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:39.411248 16456 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:57:39.411375 16456 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:57:39.411443 16456 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:57:39.411530 16456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:57:39.411623 16456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:39.411744 16456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:39.411823 16456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:57:39.411902 16456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:57:39.422597 16456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:39.424012 16456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:39.425406 16456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 20:57:39.450325 16456 server_base.cc:1034] running on GCE node
W20260923 20:57:39.450220 16463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:39.450248 16465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:39.450270 16462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:39.451941 16456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:39.453742 16456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:39.455120 16456 hybrid_clock.cc:648] HybridClock initialized: now 1790197059455044 us; error 47 us; skew 500 ppm
I20260923 20:57:39.455600 16456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:39 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:39.459859 16456 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:57:39.461971 16456 webserver.cc:460] Webserver started at http://127.0.97.62:33439/ using document root <none> and password file <none>
I20260923 20:57:39.462895 16456 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:39.463068 16456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:39.469873 16456 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:57:39.473539 16472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:39.475349 16456 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:57:39.475598 16456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "03bdb018b9c942bbac04ce5e0cde81d1"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "868c83ea3f0a3d33a5df7f99d91129d6"
server_key_iv: "4bd81f4f57efd78661f29c8e1eba2e6e"
server_key_version: "encryptionkey@0"
I20260923 20:57:39.476810 16456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:39.495337 16456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:39.501149 16456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:39.501549 16456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:39.524107 16456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39595
I20260923 20:57:39.524114 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39595 every 8 connection(s)
I20260923 20:57:39.525902 16456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:57:39.532070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16456
I20260923 20:57:39.532944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42727
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39595
--webserver_port=38967
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:57:39.534520 16524 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:57:39.538409 16524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Bootstrap starting.
I20260923 20:57:39.561445 16524 log.cc:824] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:39.575773 16524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:57:39.576470 16524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Bootstrap complete.
I20260923 20:57:39.586642 16524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:39.587662 16524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bdb018b9c942bbac04ce5e0cde81d1, State: Initialized, Role: FOLLOWER
I20260923 20:57:39.588351 16524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:39.588652 16524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:57:39.588809 16524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:57:39.589005 16524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:57:39.593431 16524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:39.594129 16524 leader_election.cc:302] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bdb018b9c942bbac04ce5e0cde81d1; no voters:
I20260923 20:57:39.594857 16524 leader_election.cc:288] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:57:39.595012 16528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:57:39.596400 16528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 LEADER]: Becoming Leader. State: Replica: 03bdb018b9c942bbac04ce5e0cde81d1, State: Running, Role: LEADER
I20260923 20:57:39.597301 16528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:57:39.597641 16524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:57:39.600492 16529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } } }
I20260923 20:57:39.600883 16529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:57:39.600108 16530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bdb018b9c942bbac04ce5e0cde81d1. Latest consensus state: current_term: 2 leader_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } } }
I20260923 20:57:39.606906 16530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:57:39.611348 16535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:57:39.623019 16535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:57:39.625306 16535 catalog_manager.cc:1259] Loaded cluster ID: 40c1a028962040f999c1f76cff68697b
I20260923 20:57:39.625582 16535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:57:39.631505 16535 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:57:39.634240 16535 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Loaded TSK: 0
I20260923 20:57:39.674436 16535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Generated new TSK 1
I20260923 20:57:39.675066 16535 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:57:39.926403 16526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:39.926780 16526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:39.926952 16526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:39.935154 16526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:39.935338 16526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:39.935518 16526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:57:39.946916 16526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38967
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:39.948379 16526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:39.949892 16526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:57:39.967568 16552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:39.967595 16551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:39.968745 16554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:39.971920 16526 server_base.cc:1034] running on GCE node
I20260923 20:57:39.973251 16526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:39.974771 16526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:39.976095 16526 hybrid_clock.cc:648] HybridClock initialized: now 1790197059976004 us; error 78 us; skew 500 ppm
I20260923 20:57:39.977136 16526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:39 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:39.981905 16526 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:57:39.984081 16526 webserver.cc:460] Webserver started at http://127.0.97.1:38967/ using document root <none> and password file <none>
I20260923 20:57:39.985092 16526 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:39.985257 16526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:39.993083 16526 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:57:39.997045 16561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:39.999279 16526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:57:39.999495 16526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "7f2cf3fee36149a79d39c74194c79df3"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "20967d945ef60746426508b80b819835"
server_key_iv: "b4016b078d5f87e97caaa9e8d3b62ba1"
server_key_version: "encryptionkey@0"
I20260923 20:57:40.000926 16526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:40.038965 16526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:40.045053 16526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:40.045375 16526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:40.046530 16526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:57:40.048054 16526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:57:40.048202 16526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:40.048333 16526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:57:40.048421 16526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:40.082702 16526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42727
I20260923 20:57:40.082767 16673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42727 every 8 connection(s)
I20260923 20:57:40.084204 16526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:57:40.091858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16526
I20260923 20:57:40.092504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37661
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39595
--webserver_port=46859
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:57:40 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:57:40.113037 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:40.087582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:43729)
0923 20:57:40.088365 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:40.088411 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:40.090630 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:40.092449 (+ 1819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:40.092459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:40.093123 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:57:40.093695 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:40.093714 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.095490 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.095496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:40.096803 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:40.096832 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.097939 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.097947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:40.098244 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:40.099051 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:57:40.099091 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:57:40.102123 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:57:40.107410 (+ 5287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:40.107424 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:40.107447 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:40.108106 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:40.108874 (+ 768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:40.108881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:40.108885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:40.109250 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:40.110580 (+ 1330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:57:40.110591 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:57:40.111085 (+ 494us) client_negotiation.cc:769] Sending connection context
0923 20:57:40.111349 (+ 264us) client_negotiation.cc:240] Negotiation successful
0923 20:57:40.111828 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":302,"threads_started":1}
I20260923 20:57:40.113580 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:40.087226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43729 (local address 127.0.97.62:39595)
0923 20:57:40.087814 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:40.087830 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:40.088571 (+ 741us) server_negotiation.cc:407] Connection header received
0923 20:57:40.091384 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:40.091397 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:40.091804 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:40.092267 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:40.093991 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.095288 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.096988 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.097647 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.097904 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:40.102520 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:40.102534 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:40.102560 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:40.102623 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:40.107163 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:40.108298 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:40.108311 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:40.108322 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:40.108616 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:40.109457 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:40.109464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:40.109467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:40.110349 (+ 882us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:40.110590 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:40.111666 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0923 20:57:40.112177 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260923 20:57:40.116612 16674 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39595
I20260923 20:57:40.116953 16674 heartbeater.cc:464] Registering TS with master...
I20260923 20:57:40.117720 16674 heartbeater.cc:511] Master 127.0.97.62:39595 requested a full tablet report, sending...
I20260923 20:57:40.120229 16489 ts_manager.cc:194] Registered new tserver with Master: 7f2cf3fee36149a79d39c74194c79df3 (127.0.97.1:42727)
I20260923 20:57:40.121879 16489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43729
W20260923 20:57:40.451314 16678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:40.451632 16678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:40.451812 16678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:40.459435 16678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:40.459578 16678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:40.459765 16678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:57:40.470055 16678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46859
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:40.471522 16678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:40.472892 16678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:57:40.487010 16686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:40.487092 16683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:40.487095 16684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:40.488158 16678 server_base.cc:1034] running on GCE node
I20260923 20:57:40.488775 16678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:40.489969 16678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:40.491245 16678 hybrid_clock.cc:648] HybridClock initialized: now 1790197060491144 us; error 103 us; skew 500 ppm
I20260923 20:57:40.491694 16678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:40 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:40.495288 16678 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:57:40.497076 16678 webserver.cc:460] Webserver started at http://127.0.97.2:46859/ using document root <none> and password file <none>
I20260923 20:57:40.497890 16678 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:40.498072 16678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:40.504112 16678 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:57:40.507318 16693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:40.508915 16678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:57:40.509130 16678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "1e86d58a30b444fda58873334b0df369"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "f65ce0b74178d9f72c1899c2cfd767f2"
server_key_iv: "a5f5a123f1f77d50640f1e98599e20a2"
server_key_version: "encryptionkey@0"
I20260923 20:57:40.510262 16678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:40.561374 16678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:40.565800 16678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:40.566172 16678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:40.567258 16678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:57:40.568722 16678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:57:40.568854 16678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:40.569000 16678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:57:40.569098 16678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:40.605033 16678 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37661
I20260923 20:57:40.605089 16805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37661 every 8 connection(s)
I20260923 20:57:40.606604 16678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:57:40.613323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16678
I20260923 20:57:40.614068 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41519
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39595
--webserver_port=39003
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:57:40 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:57:40.631271 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:40.609475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59903 (local address 127.0.97.62:39595)
0923 20:57:40.609684 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:40.609688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:40.611069 (+ 1381us) server_negotiation.cc:407] Connection header received
0923 20:57:40.613059 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:40.613065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:40.613191 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:40.613309 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:40.615610 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.616810 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.619351 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.620162 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.620333 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:40.624971 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:40.624976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:40.624980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:40.625020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:40.627799 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:40.628702 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:40.628707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:40.628710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:40.628841 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:40.629341 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:40.629345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:40.629347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:40.629652 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:40.629820 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:40.630634 (+ 814us) server_negotiation.cc:299] Negotiation successful
0923 20:57:40.630920 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:57:40.632447 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:40.610089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:59903)
0923 20:57:40.610879 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:40.610919 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:40.612709 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:40.613624 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:40.613635 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:40.614582 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:57:40.615382 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:40.615408 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.617012 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.617020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:40.619116 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:40.619137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:40.620444 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:40.620453 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:40.620797 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:40.621673 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:57:40.621719 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:57:40.624631 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:57:40.628008 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:40.628018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:40.628038 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:40.628543 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:40.628978 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:40.628983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:40.628986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:40.629203 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:40.629793 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:57:40.629801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:57:40.630228 (+ 427us) client_negotiation.cc:769] Sending connection context
0923 20:57:40.630529 (+ 301us) client_negotiation.cc:240] Negotiation successful
0923 20:57:40.631121 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":327,"threads_started":1}
I20260923 20:57:40.634346 16806 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39595
I20260923 20:57:40.634706 16806 heartbeater.cc:464] Registering TS with master...
I20260923 20:57:40.635517 16806 heartbeater.cc:511] Master 127.0.97.62:39595 requested a full tablet report, sending...
I20260923 20:57:40.637290 16489 ts_manager.cc:194] Registered new tserver with Master: 1e86d58a30b444fda58873334b0df369 (127.0.97.2:37661)
I20260923 20:57:40.638685 16489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59903
W20260923 20:57:40.963626 16809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:57:40.963932 16809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:57:40.964040 16809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:57:40.971616 16809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:57:40.971751 16809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:57:40.971868 16809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:57:40.982427 16809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39003
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:57:40.983770 16809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:57:40.985096 16809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:57:40.999599 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:40.999681 16817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:57:40.999639 16815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:57:41.001058 16809 server_base.cc:1034] running on GCE node
I20260923 20:57:41.001973 16809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:57:41.003180 16809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:57:41.004411 16809 hybrid_clock.cc:648] HybridClock initialized: now 1790197061004333 us; error 68 us; skew 500 ppm
I20260923 20:57:41.004791 16809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:57:41 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:57:41.009337 16809 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:57:41.011217 16809 webserver.cc:460] Webserver started at http://127.0.97.3:39003/ using document root <none> and password file <none>
I20260923 20:57:41.012038 16809 fs_manager.cc:359] Metadata directory not provided
I20260923 20:57:41.012226 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:57:41.018859 16809 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260923 20:57:41.022230 16824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:41.024003 16809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:57:41.024236 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "d11fdadf3b674b309540271f7b51fa80"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "d2053a1569aaa9a4449ea05d21697b77"
server_key_iv: "76a3a390e157775e41286bc7f3bd7c77"
server_key_version: "encryptionkey@0"
I20260923 20:57:41.025362 16809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:57:41.054852 16809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:57:41.059814 16809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:57:41.060165 16809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:57:41.061929 16809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:57:41.063388 16809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:57:41.063524 16809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:41.063642 16809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:57:41.063746 16809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:57:41.104967 16809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41519
I20260923 20:57:41.105033 16936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41519 every 8 connection(s)
I20260923 20:57:41.106556 16809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:57:41.111805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16809
Sep 23 20:57:41 dist-test-slave-plcg krb5kdc[15903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:57:41.125347 16674 heartbeater.cc:503] Master 127.0.97.62:39595 was elected leader, sending a full tablet report...
I20260923 20:57:41.131688 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.112893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.0.97.62:39595)
0923 20:57:41.113758 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.113763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.113778 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:57:41.113850 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.113855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.113906 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.114015 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:41.116262 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.117536 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.119641 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.120253 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.120401 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.124016 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:41.124024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:41.124029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:41.124075 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:41.128049 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.129117 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.129125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.129128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.129312 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.129713 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.129719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.129723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.130216 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:41.130515 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.130938 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.131199 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"spinlock_wait_cycles":12544,"thread_start_us":626,"threads_started":1}
I20260923 20:57:41.133086 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.109620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44427 (local address 127.0.97.62:39595)
0923 20:57:41.109808 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.109858 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.111088 (+ 1230us) server_negotiation.cc:407] Connection header received
0923 20:57:41.113785 (+ 2697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.113792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.113906 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.114015 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:41.116435 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.117536 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.120000 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.120688 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.120783 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.125209 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:41.125216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:41.125220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:41.125266 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:41.129102 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.130154 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.130157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.130159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.130272 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.130873 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.130878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.130881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.131239 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:41.131387 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.132407 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.132719 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":37120}
I20260923 20:57:41.134433 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.110113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:44427)
0923 20:57:41.110906 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.110944 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:41.113363 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:41.114382 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:41.114396 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:41.115315 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:57:41.116256 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.116280 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.117741 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.117746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.119840 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.119856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.120950 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.120957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.121292 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.121983 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:57:41.122018 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:57:41.124918 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:57:41.129283 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:41.129294 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:41.129315 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:41.130004 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:41.130440 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:57:41.130445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:57:41.130448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:57:41.130743 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:57:41.131468 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:57:41.131477 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:57:41.131993 (+ 516us) client_negotiation.cc:769] Sending connection context
0923 20:57:41.132299 (+ 306us) client_negotiation.cc:240] Negotiation successful
0923 20:57:41.132864 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":333,"threads_started":1}
I20260923 20:57:41.135937 16937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39595
I20260923 20:57:41.136236 16937 heartbeater.cc:464] Registering TS with master...
I20260923 20:57:41.136947 16937 heartbeater.cc:511] Master 127.0.97.62:39595 requested a full tablet report, sending...
I20260923 20:57:41.138392 16489 ts_manager.cc:194] Registered new tserver with Master: d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519)
I20260923 20:57:41.139187 16489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44427
I20260923 20:57:41.147660 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:57:41.155035 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.148755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.0.97.62:39595)
0923 20:57:41.148979 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.148988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.149546 (+ 558us) server_negotiation.cc:407] Connection header received
0923 20:57:41.149774 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.149779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.150038 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.150173 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:57:41.150639 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.151426 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.152938 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.153393 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.153494 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.154088 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:57:41.154318 (+ 230us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 20:57:41.154700 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60710: token expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 20:57:41.155197 16676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60710: token expired
W20260923 20:57:41.155754 16410 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:39595)
I20260923 20:57:41.169440 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.156377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.97.62:39595)
0923 20:57:41.156555 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.156559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.156575 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:57:41.156654 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.156657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.156728 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.156805 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:41.158335 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.159092 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.160613 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.161014 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.161117 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.163922 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:41.163928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:41.163931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:41.163968 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:41.166798 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.167550 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.167554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.167556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.167686 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.168099 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.168102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.168104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.168429 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:41.168621 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.168937 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.169120 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:57:41.173470 16489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:57:41.181141 16489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:57:41.218796 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.194655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.0.97.2:37661)
0923 20:57:41.208005 (+ 13350us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.208017 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.208072 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:57:41.208220 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.208232 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.208612 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.208912 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:41.210424 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.212041 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.215576 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.217376 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.217521 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.217738 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.218233 (+ 495us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.218438 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13128,"thread_start_us":12809,"threads_started":1}
I20260923 20:57:41.219894 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.194214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37661 (local address 127.0.0.1:37480)
0923 20:57:41.205060 (+ 10846us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.205103 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:41.205331 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:41.209376 (+ 4045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:41.209389 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:41.209445 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:41.210114 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.210133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.212222 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.212228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.215314 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.215330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.217640 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.217647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.217909 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.217988 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 20:57:41.218138 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 20:57:41.218351 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10551,"thread_start_us":18832,"threads_started":1}
I20260923 20:57:41.221349 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.194373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41519 (local address 127.0.0.1:49324)
0923 20:57:41.211550 (+ 17177us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.211570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:41.211716 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:41.212946 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:41.212956 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:41.212992 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:41.213985 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.214001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.216010 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.216014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.218277 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.218292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.220481 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.220488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.220672 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.220707 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:57:41.220798 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 20:57:41.220939 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17017,"mutex_wait_us":23,"thread_start_us":16809,"threads_started":1}
I20260923 20:57:41.221494 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.194835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.0.97.3:41519)
0923 20:57:41.207149 (+ 12314us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.207164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.211861 (+ 4697us) server_negotiation.cc:407] Connection header received
0923 20:57:41.211979 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.211991 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.212429 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.212669 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:41.214227 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.215824 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.218476 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.219887 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.219999 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.220246 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.220915 (+ 669us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.221121 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12036,"thread_start_us":12699,"threads_started":1}
I20260923 20:57:41.222108 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.195477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.0.97.1:42727)
0923 20:57:41.210778 (+ 15301us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.210791 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.213876 (+ 3085us) server_negotiation.cc:407] Connection header received
0923 20:57:41.214017 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.214027 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.214331 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.214558 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:41.215427 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.216757 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.219154 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.220651 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.220731 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.220960 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.221642 (+ 682us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.221862 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15100,"thread_start_us":15292,"threads_started":1}
I20260923 20:57:41.222766 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.194798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:32962)
0923 20:57:41.213663 (+ 18865us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.213684 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:41.213862 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:41.214535 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:41.214539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:41.214567 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:41.215112 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.215129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.216926 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.216929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.218954 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.218969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.221177 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.221183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.221391 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.221427 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:57:41.222377 (+ 950us) client_negotiation.cc:240] Negotiation successful
0923 20:57:41.222501 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18616,"mutex_wait_us":36,"thread_start_us":9782,"threads_started":1}
I20260923 20:57:41.224646 16741 tablet_service.cc:1467] Processing CreateTablet for tablet ca3e726871524901996d2da018e86060 (DEFAULT_TABLE table=test-table [id=f02a44e409df4f7ba4c3cfa6fbd35556]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:57:41.225292 16872 tablet_service.cc:1467] Processing CreateTablet for tablet ca3e726871524901996d2da018e86060 (DEFAULT_TABLE table=test-table [id=f02a44e409df4f7ba4c3cfa6fbd35556]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:57:41.226578 16741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3e726871524901996d2da018e86060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:57:41.226573 16609 tablet_service.cc:1467] Processing CreateTablet for tablet ca3e726871524901996d2da018e86060 (DEFAULT_TABLE table=test-table [id=f02a44e409df4f7ba4c3cfa6fbd35556]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:57:41.227337 16872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3e726871524901996d2da018e86060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:57:41.227982 16609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3e726871524901996d2da018e86060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:57:41.241672 16953 tablet_bootstrap.cc:492] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Bootstrap starting.
I20260923 20:57:41.245288 16953 tablet_bootstrap.cc:654] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Neither blocks nor log segments found. Creating new log.
I20260923 20:57:41.246814 16955 tablet_bootstrap.cc:492] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Bootstrap starting.
I20260923 20:57:41.246958 16953 log.cc:824] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:41.249145 16954 tablet_bootstrap.cc:492] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Bootstrap starting.
I20260923 20:57:41.251704 16955 tablet_bootstrap.cc:654] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Neither blocks nor log segments found. Creating new log.
I20260923 20:57:41.253134 16954 tablet_bootstrap.cc:654] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Neither blocks nor log segments found. Creating new log.
I20260923 20:57:41.253156 16955 log.cc:824] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:41.254186 16953 tablet_bootstrap.cc:492] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: No bootstrap required, opened a new log
I20260923 20:57:41.254475 16954 log.cc:824] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Log is configured to *not* fsync() on all Append() calls
I20260923 20:57:41.254608 16953 ts_tablet_manager.cc:1397] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260923 20:57:41.260846 16954 tablet_bootstrap.cc:492] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: No bootstrap required, opened a new log
I20260923 20:57:41.260846 16955 tablet_bootstrap.cc:492] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: No bootstrap required, opened a new log
I20260923 20:57:41.261186 16955 ts_tablet_manager.cc:1397] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260923 20:57:41.261186 16954 ts_tablet_manager.cc:1397] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 20:57:41.264374 16953 raft_consensus.cc:348] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.264895 16953 raft_consensus.cc:374] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:57:41.265043 16953 raft_consensus.cc:729] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11fdadf3b674b309540271f7b51fa80, State: Initialized, Role: FOLLOWER
I20260923 20:57:41.265938 16953 consensus_queue.cc:260] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.267374 16937 heartbeater.cc:503] Master 127.0.97.62:39595 was elected leader, sending a full tablet report...
I20260923 20:57:41.267962 16953 ts_tablet_manager.cc:1428] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260923 20:57:41.271384 16954 raft_consensus.cc:348] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.271931 16954 raft_consensus.cc:374] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:57:41.272089 16954 raft_consensus.cc:729] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e86d58a30b444fda58873334b0df369, State: Initialized, Role: FOLLOWER
I20260923 20:57:41.272789 16954 consensus_queue.cc:260] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.273192 16955 raft_consensus.cc:348] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.273746 16955 raft_consensus.cc:374] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:57:41.273939 16955 raft_consensus.cc:729] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2cf3fee36149a79d39c74194c79df3, State: Initialized, Role: FOLLOWER
I20260923 20:57:41.274317 16806 heartbeater.cc:503] Master 127.0.97.62:39595 was elected leader, sending a full tablet report...
I20260923 20:57:41.274645 16955 consensus_queue.cc:260] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.274991 16954 ts_tablet_manager.cc:1428] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 20:57:41.276352 16955 ts_tablet_manager.cc:1428] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260923 20:57:41.336789 16675 tablet.cc:2369] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:57:41.339062 16961 raft_consensus.cc:482] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:57:41.339362 16961 raft_consensus.cc:504] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.341553 16961 leader_election.cc:288] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e86d58a30b444fda58873334b0df369 (127.0.97.2:37661), d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519)
I20260923 20:57:41.349236 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.342012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49941 (local address 127.0.97.3:41519)
0923 20:57:41.342256 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.342265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.343047 (+ 782us) server_negotiation.cc:407] Connection header received
0923 20:57:41.343332 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.343336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.343396 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.343508 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:41.344132 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.344813 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.347309 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.348143 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.348314 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.348443 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.348810 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.348950 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:57:41.349237 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.342082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52129 (local address 127.0.97.2:37661)
0923 20:57:41.342333 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.342341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.343130 (+ 789us) server_negotiation.cc:407] Connection header received
0923 20:57:41.343280 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.343286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.343359 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.343431 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:57:41.344061 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.344789 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.347309 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.348134 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.348311 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.348439 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.348810 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.348957 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:57:41.349438 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.341910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37661 (local address 127.0.97.1:52129)
0923 20:57:41.343030 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.343044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:41.343152 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:41.343481 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:41.343489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:41.343515 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:41.343899 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.343908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.344939 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.344945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.347100 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.347116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.348413 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.348421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.348627 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.348655 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 20:57:41.348776 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 20:57:41.348950 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":54,"thread_start_us":429,"threads_started":1}
I20260923 20:57:41.349721 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.341866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41519 (local address 127.0.97.1:49941)
0923 20:57:41.342931 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.342948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:57:41.343130 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:57:41.343542 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:57:41.343547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:57:41.343577 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:57:41.343956 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.343964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.344977 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.344981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.347095 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:57:41.347113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.348452 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.348462 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:57:41.348637 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.348661 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:57:41.348777 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:57:41.348942 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":370,"thread_start_us":878,"threads_started":1}
I20260923 20:57:41.350732 16761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3e726871524901996d2da018e86060" candidate_uuid: "7f2cf3fee36149a79d39c74194c79df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e86d58a30b444fda58873334b0df369" is_pre_election: true
I20260923 20:57:41.350765 16892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3e726871524901996d2da018e86060" candidate_uuid: "7f2cf3fee36149a79d39c74194c79df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11fdadf3b674b309540271f7b51fa80" is_pre_election: true
I20260923 20:57:41.351346 16892 raft_consensus.cc:2454] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2cf3fee36149a79d39c74194c79df3 in term 0.
I20260923 20:57:41.351346 16761 raft_consensus.cc:2454] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2cf3fee36149a79d39c74194c79df3 in term 0.
I20260923 20:57:41.352192 16562 leader_election.cc:302] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e86d58a30b444fda58873334b0df369, 7f2cf3fee36149a79d39c74194c79df3, d11fdadf3b674b309540271f7b51fa80; no voters:
I20260923 20:57:41.352672 16961 raft_consensus.cc:2781] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:57:41.352837 16961 raft_consensus.cc:482] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:57:41.352962 16961 raft_consensus.cc:3037] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:57:41.355381 16961 raft_consensus.cc:504] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.356072 16961 leader_election.cc:288] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [CANDIDATE]: Term 1 election: Requested vote from peers 1e86d58a30b444fda58873334b0df369 (127.0.97.2:37661), d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519)
I20260923 20:57:41.356663 16892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3e726871524901996d2da018e86060" candidate_uuid: "7f2cf3fee36149a79d39c74194c79df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11fdadf3b674b309540271f7b51fa80"
I20260923 20:57:41.356859 16892 raft_consensus.cc:3037] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:57:41.358078 16761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3e726871524901996d2da018e86060" candidate_uuid: "7f2cf3fee36149a79d39c74194c79df3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e86d58a30b444fda58873334b0df369"
I20260923 20:57:41.358316 16761 raft_consensus.cc:3037] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 0 FOLLOWER]: Advancing to term 1
W20260923 20:57:41.358551 16938 tablet.cc:2369] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:57:41.358791 16807 tablet.cc:2369] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:57:41.359833 16892 raft_consensus.cc:2454] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2cf3fee36149a79d39c74194c79df3 in term 1.
I20260923 20:57:41.360390 16563 leader_election.cc:302] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2cf3fee36149a79d39c74194c79df3, d11fdadf3b674b309540271f7b51fa80; no voters:
I20260923 20:57:41.360786 16961 raft_consensus.cc:2781] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:57:41.361379 16961 raft_consensus.cc:686] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2cf3fee36149a79d39c74194c79df3, State: Running, Role: LEADER
I20260923 20:57:41.361532 16761 raft_consensus.cc:2454] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2cf3fee36149a79d39c74194c79df3 in term 1.
I20260923 20:57:41.361956 16961 consensus_queue.cc:237] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } }
I20260923 20:57:41.366575 16487 catalog_manager.cc:5495] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f2cf3fee36149a79d39c74194c79df3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7f2cf3fee36149a79d39c74194c79df3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f2cf3fee36149a79d39c74194c79df3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42727 } health_report { overall_health: HEALTHY } } }
I20260923 20:57:41.383741 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.377780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.0.97.1:42727)
0923 20:57:41.378009 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.378014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.378103 (+ 89us) server_negotiation.cc:407] Connection header received
0923 20:57:41.378297 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.378301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.378392 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.378525 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:57:41.378899 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.379727 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.381482 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.382019 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.382147 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.382716 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:57:41.383228 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:57:41.383348 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.383441 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.383527 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 20:57:41.392526 16761 raft_consensus.cc:1261] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 1 FOLLOWER]: Refusing update from remote peer 7f2cf3fee36149a79d39c74194c79df3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:57:41.392530 16892 raft_consensus.cc:1261] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Refusing update from remote peer 7f2cf3fee36149a79d39c74194c79df3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:57:41.393679 16964 consensus_queue.cc:1035] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11fdadf3b674b309540271f7b51fa80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:57:41.394114 16961 consensus_queue.cc:1035] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e86d58a30b444fda58873334b0df369" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:57:41.423107 16487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:57:41.423465 16487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:57:41.427562 16487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Sending DeleteTablet for 3 replicas of tablet ca3e726871524901996d2da018e86060
I20260923 20:57:41.428897 16872 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3e726871524901996d2da018e86060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:57:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49324
I20260923 20:57:41.428917 16741 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3e726871524901996d2da018e86060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:57:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37480
I20260923 20:57:41.429407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 16526
I20260923 20:57:41.430276 16609 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3e726871524901996d2da018e86060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:57:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32962
I20260923 20:57:41.440670 16978 tablet_replica.cc:330] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: stopping tablet replica
I20260923 20:57:41.441318 16979 tablet_replica.cc:330] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: stopping tablet replica
I20260923 20:57:41.441354 16978 raft_consensus.cc:2229] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:57:41.442031 16979 raft_consensus.cc:2229] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:57:41.442093 16978 raft_consensus.cc:2258] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:57:41.443100 16979 raft_consensus.cc:2258] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:57:41.443126 16977 tablet_replica.cc:330] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: stopping tablet replica
I20260923 20:57:41.443528 16977 raft_consensus.cc:2229] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:57:41.444123 16977 raft_consensus.cc:2258] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:57:41.445078 16978 ts_tablet_manager.cc:1905] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:57:41.445562 16979 ts_tablet_manager.cc:1905] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:57:41.446941 16977 ts_tablet_manager.cc:1905] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:57:41.452737 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.430958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.0.97.1:42727)
0923 20:57:41.431146 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.431152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.431168 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:57:41.431225 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.431236 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.431321 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.431430 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:41.433614 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.435249 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.437974 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.438912 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.439084 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.443376 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:41.443387 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:41.443404 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:41.443454 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:41.447847 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.448978 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.448989 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.448993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.449188 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.449675 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.449680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.449685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.450208 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:41.450529 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.452069 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.452345 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:57:41.453904 16669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:41.453972 16979 ts_tablet_manager.cc:1918] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:57:41.454150 16979 log.cc:1196] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/ca3e726871524901996d2da018e86060
I20260923 20:57:41.454319 16977 ts_tablet_manager.cc:1918] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:57:41.454502 16977 log.cc:1196] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/ca3e726871524901996d2da018e86060
I20260923 20:57:41.455185 16977 ts_tablet_manager.cc:1939] T ca3e726871524901996d2da018e86060 P d11fdadf3b674b309540271f7b51fa80: Deleting consensus metadata
I20260923 20:57:41.458225 16474 catalog_manager.cc:4841] TS d11fdadf3b674b309540271f7b51fa80 (127.0.97.3:41519): tablet ca3e726871524901996d2da018e86060 (table test-table [id=f02a44e409df4f7ba4c3cfa6fbd35556]) successfully deleted
I20260923 20:57:41.462271 16978 ts_tablet_manager.cc:1918] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:57:41.462484 16978 log.cc:1196] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/ca3e726871524901996d2da018e86060
I20260923 20:57:41.463076 16978 ts_tablet_manager.cc:1939] T ca3e726871524901996d2da018e86060 P 1e86d58a30b444fda58873334b0df369: Deleting consensus metadata
I20260923 20:57:41.465303 16473 catalog_manager.cc:4841] TS 1e86d58a30b444fda58873334b0df369 (127.0.97.2:37661): tablet ca3e726871524901996d2da018e86060 (table test-table [id=f02a44e409df4f7ba4c3cfa6fbd35556]) successfully deleted
I20260923 20:57:41.555691 16979 ts_tablet_manager.cc:1939] T ca3e726871524901996d2da018e86060 P 7f2cf3fee36149a79d39c74194c79df3: Deleting consensus metadata
I20260923 20:57:41.555971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 16526
W20260923 20:57:41.574265 16476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42727 (error 108)
I20260923 20:57:41.575018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 16678
I20260923 20:57:41.576454 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.575889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33006)
0923 20:57:41.576166 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.576283 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260923 20:57:41.576745 16476 catalog_manager.cc:4584] TS 7f2cf3fee36149a79d39c74194c79df3 (127.0.97.1:42727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca3e726871524901996d2da018e86060: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
I20260923 20:57:41.589625 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.575958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.97.2:37661)
0923 20:57:41.576169 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.576175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.576201 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:57:41.576302 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.576307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.576392 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.576520 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:41.578180 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.578920 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.580357 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.580907 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.581020 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.583940 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:41.583955 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:41.583970 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:41.584010 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:41.586872 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.587679 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.587686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.587688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.587842 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.588200 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.588203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.588204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.588586 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:41.588814 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.589166 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.589354 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 20:57:41.590438 16801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:41.630745 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.630127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33022)
0923 20:57:41.630363 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.630498 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:57:41.631999 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.631563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33026)
0923 20:57:41.631737 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.631830 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:57:41.683260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 16678
I20260923 20:57:41.693012 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.692481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33042)
0923 20:57:41.692732 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.692831 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 20:57:41.694154 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.693809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33058)
0923 20:57:41.693962 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.694031 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 20:57:41.701685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 16809
I20260923 20:57:41.716431 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.702531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.0.97.3:41519)
0923 20:57:41.702700 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:57:41.702705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:57:41.702724 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:57:41.702825 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:57:41.702829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:57:41.702894 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:57:41.702989 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:57:41.704400 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.705168 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.706607 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:57:41.707035 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:57:41.707149 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:57:41.710216 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:57:41.710234 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:57:41.710249 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:57:41.710296 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:57:41.713257 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.714172 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.714181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.714183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.714332 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:57:41.714834 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:57:41.714840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:57:41.714844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:57:41.715239 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:57:41.715497 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 20:57:41.715935 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 20:57:41.716152 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:57:41.717108 16932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:41.775024 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.774423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33066)
0923 20:57:41.774733 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.774856 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 20:57:41.776292 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.775832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33068)
0923 20:57:41.776066 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.776162 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 20:57:41.810119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 16809
I20260923 20:57:41.827188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 16456
I20260923 20:57:41.828421 16519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:57:41.956035 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:57:41.955271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42727 (local address 127.0.0.1:33080)
0923 20:57:41.955542 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 20:57:41.955666 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 20:57:41.956228 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 16456
I20260923 20:58:01.979648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39595
--webserver_interface=127.0.97.62
--webserver_port=33439
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:02.337139 16985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:02.337463 16985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:02.337652 16985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:02.345242 16985 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:58:02.345367 16985 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:58:02.345444 16985 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:58:02.345522 16985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:02.345628 16985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:02.345752 16985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:02.345880 16985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:02.345994 16985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:02.356376 16985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:02.357785 16985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:02.359272 16985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:02.372159 16993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:02.372255 16990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:02.372279 16991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:02.373332 16985 server_base.cc:1034] running on GCE node
I20260923 20:58:02.374267 16985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:02.375808 16985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:02.377086 16985 hybrid_clock.cc:648] HybridClock initialized: now 1790197082377000 us; error 62 us; skew 500 ppm
I20260923 20:58:02.377460 16985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:02 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:02.387194 16985 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:58:02.389120 16985 webserver.cc:460] Webserver started at http://127.0.97.62:33439/ using document root <none> and password file <none>
I20260923 20:58:02.390093 16985 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:02.390266 16985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:02.397027 16985 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:58:02.400637 17000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:02.402292 16985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:02.402516 16985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "03bdb018b9c942bbac04ce5e0cde81d1"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "868c83ea3f0a3d33a5df7f99d91129d6"
server_key_iv: "4bd81f4f57efd78661f29c8e1eba2e6e"
server_key_version: "encryptionkey@0"
I20260923 20:58:02.403584 16985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:02.418191 16985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:02.423082 16985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:02.423424 16985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:02.443823 16985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39595
I20260923 20:58:02.443856 17051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39595 every 8 connection(s)
I20260923 20:58:02.445516 16985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:02.453186 17052 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:58:02.455556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 16985
I20260923 20:58:02.456624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42727
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39595
--webserver_port=38967
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:58:02.457685 17052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Bootstrap starting.
I20260923 20:58:02.481699 17052 log.cc:824] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:02.498804 17052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:58:02.499393 17052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Bootstrap complete.
I20260923 20:58:02.509387 17052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:58:02.510399 17052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bdb018b9c942bbac04ce5e0cde81d1, State: Initialized, Role: FOLLOWER
I20260923 20:58:02.511171 17052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:58:02.511423 17052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:02.511545 17052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:02.511739 17052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:58:02.515976 17052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:58:02.516664 17052 leader_election.cc:302] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bdb018b9c942bbac04ce5e0cde81d1; no voters:
I20260923 20:58:02.517407 17052 leader_election.cc:288] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:58:02.517581 17056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:58:02.518843 17056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [term 3 LEADER]: Becoming Leader. State: Replica: 03bdb018b9c942bbac04ce5e0cde81d1, State: Running, Role: LEADER
I20260923 20:58:02.519575 17056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } }
I20260923 20:58:02.520038 17052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:02.522294 17058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bdb018b9c942bbac04ce5e0cde81d1. Latest consensus state: current_term: 3 leader_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } } }
I20260923 20:58:02.522691 17058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:02.523286 17057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bdb018b9c942bbac04ce5e0cde81d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39595 } } }
I20260923 20:58:02.523590 17057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:02.533654 17062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:02.543682 17062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:02.545420 17062 catalog_manager.cc:1259] Loaded cluster ID: 40c1a028962040f999c1f76cff68697b
I20260923 20:58:02.545614 17062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:02.550364 17062 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:02.552328 17062 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Loaded TSK: 0
I20260923 20:58:02.552608 17062 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03bdb018b9c942bbac04ce5e0cde81d1: Loaded TSK: 1
F20260923 20:58:02.554214 17062 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc6eaa309ad google::LogMessage::Fail() at ??:0
@ 0x7fc6eaa3388a google::LogMessage::SendToLog() at ??:0
@ 0x7fc6eaa304de google::LogMessage::Flush() at ??:0
@ 0x7fc6eaa31a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc6ee7947d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc6ee78373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc70045f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
I20260923 20:58:02.581560 17074 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fc70045e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc7004ea2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fc7004ea0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fc7005345e9 std::function<>::operator()() at ??:0
@ 0x7fc700462cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc700460c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc7004e8610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc7004e8439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc6ed67c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc6ed67f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc6ed67f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc6ed65aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc6f09346db start_thread at ??:0
@ 0x7fc6e98ca61f clone at ??:0
W20260923 20:58:02.893353 17054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:02.893661 17054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:02.893805 17054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:02.901871 17054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:02.902035 17054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:02.902253 17054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:58:02.912537 17054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38967
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:02.914068 17054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:02.915426 17054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:02.929340 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:02.929405 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:02.929455 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:02.930100 17054 server_base.cc:1034] running on GCE node
I20260923 20:58:02.930827 17054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:02.931929 17054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:02.933166 17054 hybrid_clock.cc:648] HybridClock initialized: now 1790197082933097 us; error 72 us; skew 500 ppm
I20260923 20:58:02.933501 17054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:02 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:02.936949 17054 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:58:02.938838 17054 webserver.cc:460] Webserver started at http://127.0.97.1:38967/ using document root <none> and password file <none>
I20260923 20:58:02.939608 17054 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:02.939767 17054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:02.945611 17054 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:58:02.948668 17109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:02.950278 17054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:02.950490 17054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "7f2cf3fee36149a79d39c74194c79df3"
format_stamp: "Formatted at 2026-09-23 20:57:16 on dist-test-slave-plcg"
server_key: "20967d945ef60746426508b80b819835"
server_key_iv: "b4016b078d5f87e97caaa9e8d3b62ba1"
server_key_version: "encryptionkey@0"
I20260923 20:58:02.951576 17054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:02.969452 17054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:02.974283 17054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:02.974593 17054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:02.975759 17054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:02.977265 17054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:02.977409 17054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:02.977557 17054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:02.977655 17054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:03.013187 17054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42727
I20260923 20:58:03.013236 17221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42727 every 8 connection(s)
I20260923 20:58:03.014817 17054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:58:03.016659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 17054
I20260923 20:58:03.017428 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37661
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39595
--webserver_port=46859
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:58:03.020491 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.018449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:59923)
0923 20:58:03.019331 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.019540 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":346,"threads_started":1}
W20260923 20:58:03.021276 17222 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39595 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39595: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
I20260923 20:58:03.022347 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.021938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:57087)
0923 20:58:03.022120 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.022200 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:58:03.023473 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.022991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:55187)
0923 20:58:03.023186 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.023295 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 20:58:03.023789 17222 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:58:03.364573 17225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:03.364886 17225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:03.364985 17225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:03.372509 17225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:03.372648 17225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:03.372767 17225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:58:03.383040 17225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46859
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:03.384375 17225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:03.385686 17225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:03.399502 17231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:03.399586 17233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:03.399586 17230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:03.400722 17225 server_base.cc:1034] running on GCE node
I20260923 20:58:03.401459 17225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:03.402601 17225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:03.403829 17225 hybrid_clock.cc:648] HybridClock initialized: now 1790197083403779 us; error 46 us; skew 500 ppm
I20260923 20:58:03.404174 17225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:03 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:03.407665 17225 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:58:03.409323 17225 webserver.cc:460] Webserver started at http://127.0.97.2:46859/ using document root <none> and password file <none>
I20260923 20:58:03.410175 17225 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:03.410331 17225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:03.416330 17225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:58:03.419582 17240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:03.421175 17225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:58:03.421404 17225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "1e86d58a30b444fda58873334b0df369"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "f65ce0b74178d9f72c1899c2cfd767f2"
server_key_iv: "a5f5a123f1f77d50640f1e98599e20a2"
server_key_version: "encryptionkey@0"
I20260923 20:58:03.422641 17225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:03.449378 17225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:03.454298 17225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:03.454608 17225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:03.455771 17225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:03.457481 17225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:03.457621 17225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:03.457770 17225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:03.457896 17225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 20:58:03.493906 17225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37661
I20260923 20:58:03.494027 17352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37661 every 8 connection(s)
I20260923 20:58:03.495487 17225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:58:03.500998 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.498900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:51543)
0923 20:58:03.499808 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.500054 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":318,"threads_started":1}
W20260923 20:58:03.501850 17353 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39595 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39595: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
I20260923 20:58:03.502827 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.502456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:54981)
0923 20:58:03.502624 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.502708 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 20:58:03.503708 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.503352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:46423)
0923 20:58:03.503501 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.503586 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:58:03.503944 17353 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:58:03.504808 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 17225
I20260923 20:58:03.505434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41519
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39595
--webserver_port=39003
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:03.849537 17356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:03.849928 17356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:03.850129 17356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:03.858172 17356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:03.858350 17356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:03.858542 17356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:58:03.868844 17356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39003
--tserver_master_addrs=127.0.97.62:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:03.870366 17356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:03.871752 17356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:03.885941 17361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:03.885994 17362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:03.886142 17364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:03.887889 17356 server_base.cc:1034] running on GCE node
I20260923 20:58:03.888818 17356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:03.890254 17356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:03.891755 17356 hybrid_clock.cc:648] HybridClock initialized: now 1790197083891608 us; error 137 us; skew 500 ppm
I20260923 20:58:03.892179 17356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:03 dist-test-slave-plcg krb5kdc[15903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:03.896049 17356 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:58:03.897689 17356 webserver.cc:460] Webserver started at http://127.0.97.3:39003/ using document root <none> and password file <none>
I20260923 20:58:03.898572 17356 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:03.898739 17356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:03.904786 17356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:58:03.907810 17371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:03.909358 17356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:58:03.909564 17356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "d11fdadf3b674b309540271f7b51fa80"
format_stamp: "Formatted at 2026-09-23 20:57:17 on dist-test-slave-plcg"
server_key: "d2053a1569aaa9a4449ea05d21697b77"
server_key_iv: "76a3a390e157775e41286bc7f3bd7c77"
server_key_version: "encryptionkey@0"
I20260923 20:58:03.910737 17356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:03.938050 17356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:03.942962 17356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:03.943284 17356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:03.944391 17356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:03.945801 17356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:03.945983 17356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:03.946128 17356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:03.946228 17356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:03.982342 17356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41519
I20260923 20:58:03.982435 17483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41519 every 8 connection(s)
I20260923 20:58:03.983935 17356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:58:03.989208 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.987284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:57187)
0923 20:58:03.988055 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.988285 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":305,"threads_started":1}
W20260923 20:58:03.989985 17484 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39595 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39595: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
I20260923 20:58:03.990883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 17356
I20260923 20:58:03.991024 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.990556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:59481)
0923 20:58:03.990760 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.990857 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 20:58:03.992017 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:03.991647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:45797)
0923 20:58:03.991808 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:03.991895 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 20:58:03.992347 17484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:58:04.025403 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:04.024530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:57285)
0923 20:58:04.025056 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:04.025155 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":267,"threads_started":1}
I20260923 20:58:04.505496 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:04.504628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:41225)
0923 20:58:04.505150 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:04.505244 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":259,"threads_started":1}
I20260923 20:58:04.993948 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:04.992968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:54687)
0923 20:58:04.993519 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:04.993620 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260923 20:58:05.027429 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:05.026635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:35369)
0923 20:58:05.027155 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:05.027258 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":229,"threads_started":1}
I20260923 20:58:05.507335 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:05.506589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:55857)
0923 20:58:05.507068 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:05.507160 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":218,"threads_started":1}
I20260923 20:58:05.995797 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:05.994968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:53203)
0923 20:58:05.995506 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:05.995622 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260923 20:58:06.029230 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:06.028476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:52987)
0923 20:58:06.028950 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:06.029043 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260923 20:58:06.509085 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:06.508283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:44741)
0923 20:58:06.508820 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:06.508903 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":322,"threads_started":1}
I20260923 20:58:06.997717 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:06.996946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:43359)
0923 20:58:06.997424 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:06.997513 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260923 20:58:07.031155 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:07.030372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:49727)
0923 20:58:07.030859 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:07.030945 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260923 20:58:07.511188 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:07.510287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:57297)
0923 20:58:07.510822 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:07.510924 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":258,"threads_started":1}
I20260923 20:58:07.999727 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:07.998819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:36559)
0923 20:58:07.999430 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:07.999541 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
I20260923 20:58:08.033056 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:08.032204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:37657)
0923 20:58:08.032775 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:08.032870 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":254,"threads_started":1}
I20260923 20:58:08.513180 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:08.512268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:55451)
0923 20:58:08.512853 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:08.512946 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":329,"threads_started":1}
I20260923 20:58:09.001513 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:09.000690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:45493)
0923 20:58:09.001247 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:09.001333 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":299,"threads_started":1}
I20260923 20:58:09.034770 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:09.034040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:39207)
0923 20:58:09.034505 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:09.034601 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260923 20:58:09.515167 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:09.514298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:39455)
0923 20:58:09.514784 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:09.514972 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260923 20:58:10.003338 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:10.002510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:58947)
0923 20:58:10.003029 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:10.003157 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":271,"threads_started":1}
I20260923 20:58:10.036720 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:10.035858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:38977)
0923 20:58:10.036393 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:10.036484 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":230,"threads_started":1}
I20260923 20:58:10.517244 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:10.516266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:52661)
0923 20:58:10.516884 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:10.516999 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":275,"threads_started":1}
I20260923 20:58:11.005330 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:11.004457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:48505)
0923 20:58:11.004985 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:11.005080 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":234,"threads_started":1}
I20260923 20:58:11.038995 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:11.037933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:37893)
0923 20:58:11.038729 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:11.038826 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":374,"threads_started":1}
I20260923 20:58:11.519028 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:11.518311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:59375)
0923 20:58:11.518774 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:11.518863 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":213,"threads_started":1}
I20260923 20:58:12.007264 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:12.006479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:56867)
0923 20:58:12.006985 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:12.007084 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":254,"threads_started":1}
I20260923 20:58:12.041020 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:12.040137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:32777)
0923 20:58:12.040751 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:12.040840 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":332,"threads_started":1}
I20260923 20:58:12.520721 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:12.519942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:34221)
0923 20:58:12.520452 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:12.520534 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260923 20:58:13.009124 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:13.008319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:38043)
0923 20:58:13.008845 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:13.008935 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":278,"threads_started":1}
I20260923 20:58:13.042865 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:13.042123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:45609)
0923 20:58:13.042610 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:13.042698 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260923 20:58:13.522621 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:13.521781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:60351)
0923 20:58:13.522369 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:13.522449 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":302,"threads_started":1}
I20260923 20:58:14.010993 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:14.010226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:37047)
0923 20:58:14.010728 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:14.010821 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":266,"threads_started":1}
I20260923 20:58:14.044721 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:14.043870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:39485)
0923 20:58:14.044418 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:14.044532 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":311,"threads_started":1}
I20260923 20:58:14.524735 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:14.523697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:60801)
0923 20:58:14.524381 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:14.524501 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":279,"threads_started":1}
I20260923 20:58:15.013003 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:15.012199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:56049)
0923 20:58:15.012700 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:15.012798 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":239,"threads_started":1}
I20260923 20:58:15.046568 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:15.045775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:50645)
0923 20:58:15.046321 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:15.046407 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":262,"threads_started":1}
I20260923 20:58:15.526837 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:15.525927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:60845)
0923 20:58:15.526439 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:15.526622 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":307,"threads_started":1}
I20260923 20:58:16.015003 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:16.014194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:54099)
0923 20:58:16.014755 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:16.014853 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":280,"threads_started":1}
I20260923 20:58:16.048391 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:16.047583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:58129)
0923 20:58:16.048075 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:16.048173 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260923 20:58:16.528455 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:16.527740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:38507)
0923 20:58:16.528180 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:16.528265 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":216,"threads_started":1}
I20260923 20:58:17.016922 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:17.015959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:43631)
0923 20:58:17.016557 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:17.016678 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":274,"threads_started":1}
I20260923 20:58:17.050283 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:17.049388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:49839)
0923 20:58:17.049996 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:17.050095 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":324,"threads_started":1}
I20260923 20:58:17.530223 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:17.529396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:35037)
0923 20:58:17.529949 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:17.530029 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":285,"threads_started":1}
I20260923 20:58:18.018860 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:18.018025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:45675)
0923 20:58:18.018573 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:18.018661 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":216,"threads_started":1}
I20260923 20:58:18.052259 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:18.051477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.1:46787)
0923 20:58:18.051974 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:18.052070 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260923 20:58:18.532420 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:18.531542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.2:50065)
0923 20:58:18.532128 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:18.532231 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":338,"threads_started":1}
I20260923 20:58:18.996675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 17054
I20260923 20:58:19.015856 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:18.997899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.0.97.1:42727)
0923 20:58:18.998565 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:18.998583 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:18.998857 (+ 274us) server_negotiation.cc:407] Connection header received
0923 20:58:19.000585 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:19.000595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:19.001055 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:19.001369 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:19.002959 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:19.004257 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:19.005582 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:19.006374 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:19.006526 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:19.009284 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:19.009293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:19.009312 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:19.009353 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:19.012328 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:19.013088 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:19.013095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:19.013102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:19.013274 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:19.013598 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:19.013601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:19.013603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:19.014221 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:19.014450 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:19.014940 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 20:58:19.015411 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":344,"threads_started":1}
I20260923 20:58:19.017511 17217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:19.021060 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:19.020169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39595 (local address 127.0.97.3:40069)
0923 20:58:19.020696 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:19.020813 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
I20260923 20:58:19.103735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 17054
I20260923 20:58:19.121446 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 17225
I20260923 20:58:19.141141 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:19.122603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.0.97.2:37661)
0923 20:58:19.123178 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:19.123193 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:19.123235 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:58:19.124938 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:19.124950 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:19.125436 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:19.126060 (+ 624us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:19.127617 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:19.128938 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:19.130268 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:19.131023 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:19.131184 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:19.133900 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:19.133916 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:19.133943 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:19.134013 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:19.137089 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:19.138399 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:19.138407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:19.138415 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:19.138618 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:19.139020 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:19.139023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:19.139025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:19.139577 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:19.139841 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:19.140332 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 20:58:19.140837 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":250,"threads_started":1}
I20260923 20:58:19.142858 17348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:19.230099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 17225
I20260923 20:58:19.248183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 17356
I20260923 20:58:19.266718 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:19.249172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.0.97.3:41519)
0923 20:58:19.249792 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:19.249807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:19.249913 (+ 106us) server_negotiation.cc:407] Connection header received
0923 20:58:19.251525 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:19.251536 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:19.251979 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:19.252294 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:19.253719 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:19.255049 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:19.256299 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:19.256994 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:19.257148 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:19.259879 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:19.259888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:19.259905 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:19.259949 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:19.263157 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:19.263975 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:19.263984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:19.263991 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:19.264187 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:19.264540 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:19.264543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:19.264545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:19.265135 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:19.265369 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:19.266010 (+ 641us) server_negotiation.cc:299] Negotiation successful
0923 20:58:19.266403 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":315,"threads_started":1}
I20260923 20:58:19.268146 17479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:19.353530 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 17356
2026-09-23T20:58:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65062 ms)
[----------] 30 tests from SecurityITest (308323 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:19 dist-test-slave-plcg krb5kdc[17542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:19 dist-test-slave-plcg krb5kdc[17542](info): set up 2 sockets
Sep 23 20:58:19 dist-test-slave-plcg krb5kdc[17542](info): commencing operation
krb5kdc: starting...
W20260923 20:58:20.470096 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:20 dist-test-slave-plcg krb5kdc[17542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:58:20.730047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44203
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:21.072384 17555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:21.072702 17555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:21.072798 17555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:21.080209 17555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:21.080355 17555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:21.080403 17555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:21.080458 17555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:21.080504 17555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:21.090821 17555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:21.092183 17555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:21.093488 17555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:21.106276 17561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:21.106348 17560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:21.106389 17563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:21.107447 17555 server_base.cc:1034] running on GCE node
I20260923 20:58:21.108327 17555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:21.109942 17555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:21.111212 17555 hybrid_clock.cc:648] HybridClock initialized: now 1790197101111154 us; error 41 us; skew 500 ppm
I20260923 20:58:21.111639 17555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:21 dist-test-slave-plcg krb5kdc[17542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:21.115136 17555 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:58:21.116784 17555 webserver.cc:460] Webserver started at http://127.0.97.62:45229/ using document root <none> and password file <none>
I20260923 20:58:21.117568 17555 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:21.117726 17555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:21.118117 17555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:21.121264 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "37a14446f4ad4c3bb04df12c4627f6a3"
format_stamp: "Formatted at 2026-09-23 20:58:21 on dist-test-slave-plcg"
server_key: "095dfa73daaa263b0f9602a450a7cfd7"
server_key_iv: "d1862026843dd7a64cbd523898da2204"
server_key_version: "encryptionkey@0"
I20260923 20:58:21.122182 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "37a14446f4ad4c3bb04df12c4627f6a3"
format_stamp: "Formatted at 2026-09-23 20:58:21 on dist-test-slave-plcg"
server_key: "095dfa73daaa263b0f9602a450a7cfd7"
server_key_iv: "d1862026843dd7a64cbd523898da2204"
server_key_version: "encryptionkey@0"
I20260923 20:58:21.127776 17555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:58:21.131644 17570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:21.133072 17555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:58:21.133282 17555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "37a14446f4ad4c3bb04df12c4627f6a3"
format_stamp: "Formatted at 2026-09-23 20:58:21 on dist-test-slave-plcg"
server_key: "095dfa73daaa263b0f9602a450a7cfd7"
server_key_iv: "d1862026843dd7a64cbd523898da2204"
server_key_version: "encryptionkey@0"
I20260923 20:58:21.133499 17555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:21.168519 17555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:21.173220 17555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:21.173554 17555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:21.193504 17555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44203
I20260923 20:58:21.193534 17621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44203 every 8 connection(s)
I20260923 20:58:21.195058 17555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:21.195933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 17555
I20260923 20:58:21.196148 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:21.197762 388 external_mini_cluster.cc:1428] Setting key 2377d059f0800c1125bc288e7a8de5fd
I20260923 20:58:21.200634 17622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:58:21 dist-test-slave-plcg krb5kdc[17542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:21.215776 17622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3: Bootstrap starting.
I20260923 20:58:21.219255 17622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:21.220351 17622 log.cc:824] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:21.223480 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:21.200580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.0.97.62:44203)
0923 20:58:21.201554 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:21.201566 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:21.201610 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:58:21.203353 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:21.203378 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:21.203829 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:21.204216 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:21.206232 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:21.207692 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:21.209630 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:21.210471 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:21.210620 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:21.215608 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:21.215617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:21.215634 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:21.215683 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:21.219010 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:21.219887 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:21.219896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:21.219903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:21.220077 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:21.220515 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:21.220522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:21.220525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:21.221125 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:21.221461 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:21.221987 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 20:58:21.222453 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":444,"threads_started":1}
I20260923 20:58:21.223660 17622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3: No bootstrap required, opened a new log
I20260923 20:58:21.232666 17622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44203 } }
I20260923 20:58:21.233110 17622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:21.233227 17622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a14446f4ad4c3bb04df12c4627f6a3, State: Initialized, Role: FOLLOWER
I20260923 20:58:21.233889 17622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44203 } }
I20260923 20:58:21.234155 17622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:21.234310 17622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:21.234490 17622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:21.237347 17622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44203 } }
I20260923 20:58:21.237987 17622 leader_election.cc:302] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a14446f4ad4c3bb04df12c4627f6a3; no voters:
I20260923 20:58:21.238631 17622 leader_election.cc:288] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:21.238778 17627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:21.239470 17627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [term 1 LEADER]: Becoming Leader. State: Replica: 37a14446f4ad4c3bb04df12c4627f6a3, State: Running, Role: LEADER
I20260923 20:58:21.240079 17627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44203 } }
I20260923 20:58:21.240985 17622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:21.243400 17629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a14446f4ad4c3bb04df12c4627f6a3. Latest consensus state: current_term: 1 leader_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44203 } } }
I20260923 20:58:21.244310 17629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:21.243780 17628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a14446f4ad4c3bb04df12c4627f6a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44203 } } }
I20260923 20:58:21.245206 17628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:21.246012 17636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:21.250962 17636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:21.261328 17636 catalog_manager.cc:1347] Generated new cluster ID: 494d3a9411df44129048ee6b4172608e
I20260923 20:58:21.261487 17636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:21.300117 17636 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:21.301584 17636 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:21.308212 17636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3: Generated new TSK 0
I20260923 20:58:21.309098 17636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:58:21.381520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44203
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:21.728858 17648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:21.729156 17648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:21.729323 17648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:21.737151 17648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:21.737308 17648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:21.737512 17648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:58:21.748401 17648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:21.749778 17648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:21.751250 17648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:21.765410 17656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:21.765314 17654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:21.765437 17653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:21.766484 17648 server_base.cc:1034] running on GCE node
I20260923 20:58:21.767092 17648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:21.768059 17648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:21.769295 17648 hybrid_clock.cc:648] HybridClock initialized: now 1790197101769231 us; error 51 us; skew 500 ppm
I20260923 20:58:21.769623 17648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:21 dist-test-slave-plcg krb5kdc[17542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:21.772984 17648 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:58:21.774593 17648 webserver.cc:460] Webserver started at http://127.0.97.1:46065/ using document root <none> and password file <none>
I20260923 20:58:21.775285 17648 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:21.775398 17648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:21.775671 17648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:21.778831 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "29abe1996b164974b34bf2c7336c1b34"
format_stamp: "Formatted at 2026-09-23 20:58:21 on dist-test-slave-plcg"
server_key: "b28d29469be200db093a51553eb9a053"
server_key_iv: "0d01e32c42c71f38a90688f84b05ac84"
server_key_version: "encryptionkey@0"
I20260923 20:58:21.779649 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "29abe1996b164974b34bf2c7336c1b34"
format_stamp: "Formatted at 2026-09-23 20:58:21 on dist-test-slave-plcg"
server_key: "b28d29469be200db093a51553eb9a053"
server_key_iv: "0d01e32c42c71f38a90688f84b05ac84"
server_key_version: "encryptionkey@0"
I20260923 20:58:21.785271 17648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:58:21.789170 17663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:21.790722 17648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:58:21.790886 17648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "29abe1996b164974b34bf2c7336c1b34"
format_stamp: "Formatted at 2026-09-23 20:58:21 on dist-test-slave-plcg"
server_key: "b28d29469be200db093a51553eb9a053"
server_key_iv: "0d01e32c42c71f38a90688f84b05ac84"
server_key_version: "encryptionkey@0"
I20260923 20:58:21.791077 17648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:21.823791 17648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:21.828214 17648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:21.828490 17648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:21.829455 17648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:21.830904 17648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:21.830983 17648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:21.831076 17648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:21.831121 17648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:21.866149 17648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40439
I20260923 20:58:21.866185 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40439 every 8 connection(s)
I20260923 20:58:21.867549 17648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:58:21.868546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 17648
I20260923 20:58:21.868752 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:58:21.869433 388 external_mini_cluster.cc:1428] Setting key 98a7036cb1c82af123107b7f14938a79
Sep 23 20:58:21 dist-test-slave-plcg krb5kdc[17542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:21.896924 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:21.870350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43021 (local address 127.0.97.62:44203)
0923 20:58:21.875550 (+ 5200us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:21.875555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:21.875574 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:58:21.877872 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:21.877882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:21.877955 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:21.878054 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:21.880413 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:21.881666 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:21.884057 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:21.884874 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:21.885031 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:21.890513 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:21.890519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:21.890522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:21.890562 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:21.893177 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:21.894123 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:21.894129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:21.894133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:21.894309 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:21.894933 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:21.894938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:21.894942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:21.895238 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:21.895418 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:21.896263 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 20:58:21.896541 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5086,"thread_start_us":4972,"threads_started":1}
I20260923 20:58:21.898082 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:21.871064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44203 (local address 127.0.97.1:43021)
0923 20:58:21.874539 (+ 3475us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:21.874613 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:21.877468 (+ 2855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:21.878382 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:21.878398 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:21.879267 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:21.880165 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:21.880196 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:21.881914 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:21.881918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:21.883836 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:21.883854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:21.885246 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:21.885257 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:21.885636 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:21.886580 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:21.886638 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:21.890213 (+ 3575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:21.893373 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:21.893383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:21.893403 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:21.893975 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:21.894447 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:21.894452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:21.894456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:21.894789 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:21.895402 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:21.895410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:21.895835 (+ 425us) client_negotiation.cc:769] Sending connection context
0923 20:58:21.896147 (+ 312us) client_negotiation.cc:240] Negotiation successful
0923 20:58:21.896747 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3092,"thread_start_us":2823,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:58:21.899806 17776 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44203
I20260923 20:58:21.900137 17776 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:21.901021 17776 heartbeater.cc:511] Master 127.0.97.62:44203 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 20:58:21.903537 17587 ts_manager.cc:194] Registered new tserver with Master: 29abe1996b164974b34bf2c7336c1b34 (127.0.97.1:40439)
I20260923 20:58:21.905040 17587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:58:21.933153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44203
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:22.280057 17782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:22.280380 17782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:22.280555 17782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:22.288168 17782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:22.288316 17782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:22.288534 17782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:58:22.299147 17782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:22.300437 17782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:22.301730 17782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:22.315886 17788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:22.315956 17787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:22.315932 17790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:22.316885 17782 server_base.cc:1034] running on GCE node
I20260923 20:58:22.317631 17782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:22.318749 17782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:22.319957 17782 hybrid_clock.cc:648] HybridClock initialized: now 1790197102319892 us; error 61 us; skew 500 ppm
I20260923 20:58:22.320290 17782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:22 dist-test-slave-plcg krb5kdc[17542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:22.323782 17782 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:58:22.325425 17782 webserver.cc:460] Webserver started at http://127.0.97.2:36903/ using document root <none> and password file <none>
I20260923 20:58:22.326253 17782 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:22.326395 17782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:22.326731 17782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:22.329934 17782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "70370753c9bb413e8051a370cc94ca99"
format_stamp: "Formatted at 2026-09-23 20:58:22 on dist-test-slave-plcg"
server_key: "544278ce9444a0f7e67c26bc4ff09a67"
server_key_iv: "97fa3b3bec60c390f8190516664318ea"
server_key_version: "encryptionkey@0"
I20260923 20:58:22.330868 17782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "70370753c9bb413e8051a370cc94ca99"
format_stamp: "Formatted at 2026-09-23 20:58:22 on dist-test-slave-plcg"
server_key: "544278ce9444a0f7e67c26bc4ff09a67"
server_key_iv: "97fa3b3bec60c390f8190516664318ea"
server_key_version: "encryptionkey@0"
I20260923 20:58:22.336438 17782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:58:22.340284 17797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:22.341864 17782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:22.342087 17782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "70370753c9bb413e8051a370cc94ca99"
format_stamp: "Formatted at 2026-09-23 20:58:22 on dist-test-slave-plcg"
server_key: "544278ce9444a0f7e67c26bc4ff09a67"
server_key_iv: "97fa3b3bec60c390f8190516664318ea"
server_key_version: "encryptionkey@0"
I20260923 20:58:22.342304 17782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:22.366194 17782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:22.372082 17782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:22.372350 17782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:22.373345 17782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:22.374819 17782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:22.374897 17782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:22.374986 17782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:22.375030 17782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:22.409771 17782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33293
I20260923 20:58:22.409896 17909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33293 every 8 connection(s)
I20260923 20:58:22.411180 17782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:58:22.420715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 17782
I20260923 20:58:22.420915 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:58:22.421388 388 external_mini_cluster.cc:1428] Setting key 7e6852e4be6e8addcc560c9665dab04d
Sep 23 20:58:22 dist-test-slave-plcg krb5kdc[17542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:22.432878 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:22.413972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43313 (local address 127.0.97.62:44203)
0923 20:58:22.414472 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:22.414476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:22.415339 (+ 863us) server_negotiation.cc:407] Connection header received
0923 20:58:22.417290 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:22.417296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:22.417396 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:22.417532 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:22.419238 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.419968 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.421790 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.422450 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.422596 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:22.426658 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:22.426663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:22.426667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:22.426704 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:22.429481 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:22.430411 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:22.430416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:22.430419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:22.430563 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:22.431047 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:22.431051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:22.431053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:22.431325 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:22.431502 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:22.432235 (+ 733us) server_negotiation.cc:299] Negotiation successful
0923 20:58:22.432473 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":288,"threads_started":1}
I20260923 20:58:22.433988 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:22.414425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44203 (local address 127.0.97.2:43313)
0923 20:58:22.415164 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:22.415209 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:22.416982 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:22.417791 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:22.417802 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:22.418438 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:22.419025 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:22.419058 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.420150 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.420154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:22.421631 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:22.421647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.422749 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.422756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:22.423005 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:22.423785 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:22.423814 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:22.426381 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:22.429659 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:22.429670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:22.429690 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:22.430259 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:22.430692 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:22.430697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:22.430701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:22.430911 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:22.431465 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:22.431474 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:22.431855 (+ 381us) client_negotiation.cc:769] Sending connection context
0923 20:58:22.432132 (+ 277us) client_negotiation.cc:240] Negotiation successful
0923 20:58:22.432724 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":288,"threads_started":1}
I20260923 20:58:22.435678 17910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44203
I20260923 20:58:22.436000 17910 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:22.436796 17910 heartbeater.cc:511] Master 127.0.97.62:44203 requested a full tablet report, sending...
I20260923 20:58:22.438571 17587 ts_manager.cc:194] Registered new tserver with Master: 70370753c9bb413e8051a370cc94ca99 (127.0.97.2:33293)
I20260923 20:58:22.439888 17587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43313
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:58:22.481356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44203
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:22.820844 17916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:22.821152 17916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:22.821259 17916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:22.828853 17916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:22.828980 17916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:22.829097 17916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:58:22.839291 17916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:22.840655 17916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:22.842110 17916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:22.856065 17921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:22.856150 17924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:22.856109 17922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:22.857025 17916 server_base.cc:1034] running on GCE node
I20260923 20:58:22.857723 17916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:22.858737 17916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:22.859972 17916 hybrid_clock.cc:648] HybridClock initialized: now 1790197102859902 us; error 60 us; skew 500 ppm
I20260923 20:58:22.860339 17916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:22 dist-test-slave-plcg krb5kdc[17542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:22.869418 17916 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:58:22.871197 17916 webserver.cc:460] Webserver started at http://127.0.97.3:32917/ using document root <none> and password file <none>
I20260923 20:58:22.871934 17916 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:22.872093 17916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:22.872418 17916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:22.875643 17916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "d564e5c552fc45f9a49db782af8a8c7f"
format_stamp: "Formatted at 2026-09-23 20:58:22 on dist-test-slave-plcg"
server_key: "2ba8ba76af0aca7eba665ba9d5ab79e5"
server_key_iv: "026ad613d72ec696311d14942c560570"
server_key_version: "encryptionkey@0"
I20260923 20:58:22.876530 17916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d564e5c552fc45f9a49db782af8a8c7f"
format_stamp: "Formatted at 2026-09-23 20:58:22 on dist-test-slave-plcg"
server_key: "2ba8ba76af0aca7eba665ba9d5ab79e5"
server_key_iv: "026ad613d72ec696311d14942c560570"
server_key_version: "encryptionkey@0"
I20260923 20:58:22.882103 17916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:58:22.886013 17931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:22.887457 17916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:22.887668 17916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "d564e5c552fc45f9a49db782af8a8c7f"
format_stamp: "Formatted at 2026-09-23 20:58:22 on dist-test-slave-plcg"
server_key: "2ba8ba76af0aca7eba665ba9d5ab79e5"
server_key_iv: "026ad613d72ec696311d14942c560570"
server_key_version: "encryptionkey@0"
I20260923 20:58:22.887934 17916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:22.908313 17776 heartbeater.cc:503] Master 127.0.97.62:44203 was elected leader, sending a full tablet report...
I20260923 20:58:22.916556 17916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:22.921262 17916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:22.921581 17916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:22.922709 17916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:22.924139 17916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:22.924253 17916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:22.924407 17916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:22.924482 17916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:22.959442 17916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46069
I20260923 20:58:22.959537 18043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46069 every 8 connection(s)
I20260923 20:58:22.960848 17916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:58:22.966995 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 17916
I20260923 20:58:22.967213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:58:22.967702 388 external_mini_cluster.cc:1428] Setting key 0182905c8520e054904c7183ff8153cf
Sep 23 20:58:22 dist-test-slave-plcg krb5kdc[17542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:22.982527 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:22.964025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57277 (local address 127.0.97.62:44203)
0923 20:58:22.964510 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:22.964514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:22.965392 (+ 878us) server_negotiation.cc:407] Connection header received
0923 20:58:22.967511 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:22.967516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:22.967629 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:22.967752 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:22.969377 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.970220 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.971850 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.972243 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.972383 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:22.976188 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:22.976193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:22.976197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:22.976243 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:22.978849 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:22.979704 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:22.979708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:22.979710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:22.979841 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:22.980276 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:22.980279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:22.980281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:22.980548 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:22.980671 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:22.981667 (+ 996us) server_negotiation.cc:299] Negotiation successful
0923 20:58:22.982018 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260923 20:58:22.983626 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:22.964453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44203 (local address 127.0.97.3:57277)
0923 20:58:22.965214 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:22.965254 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:22.967162 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:22.968019 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:22.968029 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:22.968577 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:22.969189 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:22.969207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.970389 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.970394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:22.971661 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:22.971680 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:22.972485 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:22.972492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:22.972739 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:22.973415 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:22.973438 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:22.975910 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:22.979030 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:22.979039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:22.979055 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:22.979559 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:22.979943 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:22.979947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:22.979956 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:22.980174 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:22.980685 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:22.980693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:22.981072 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 20:58:22.981520 (+ 448us) client_negotiation.cc:240] Negotiation successful
0923 20:58:22.982312 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":309,"threads_started":1}
I20260923 20:58:22.985011 18044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44203
I20260923 20:58:22.985292 18044 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:22.986083 18044 heartbeater.cc:511] Master 127.0.97.62:44203 requested a full tablet report, sending...
I20260923 20:58:22.987550 17587 ts_manager.cc:194] Registered new tserver with Master: d564e5c552fc45f9a49db782af8a8c7f (127.0.97.3:46069)
I20260923 20:58:22.988471 17587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57277
I20260923 20:58:22.996330 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:58:23.014407 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.001243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.97.62:44203)
0923 20:58:23.001435 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.001439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.001924 (+ 485us) server_negotiation.cc:407] Connection header received
0923 20:58:23.002088 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.002092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.002160 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.002247 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:23.003723 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.004451 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.006070 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.006472 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.006574 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.009189 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:23.009194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:23.009198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:23.009235 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:23.011929 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.012645 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.012650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.012652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.012789 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.013115 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.013119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.013120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.013433 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:23.013614 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.013975 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.014179 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:58:23.018596 17587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:58:23.025887 17587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:58:23.051829 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.038694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46069 (local address 127.0.0.1:44280)
0923 20:58:23.039453 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:23.039505 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:23.039700 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:23.040917 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:23.040930 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:23.040973 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:23.041403 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.041419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.044018 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.044028 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.047407 (+ 3379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.047436 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.050428 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.050435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.050685 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.050802 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 20:58:23.050996 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 20:58:23.051295 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":299,"threads_started":1}
I20260923 20:58:23.053287 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.038915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.0.97.3:46069)
0923 20:58:23.039698 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.039710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.039750 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:58:23.040009 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.040018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.040294 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.040508 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:23.041675 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.043782 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.047706 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.049723 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.049994 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.050293 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.052541 (+ 2248us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.052822 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":1069,"threads_started":1}
I20260923 20:58:23.057330 17979 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea2c0aef524c71bf00997946bcafd4 (DEFAULT_TABLE table=test-table [id=90d85d05bdf04e4ea5adeb3c38d69414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:23.059034 17979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea2c0aef524c71bf00997946bcafd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:23.060657 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.039858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.0.97.2:33293)
0923 20:58:23.050278 (+ 10420us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.050289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.051194 (+ 905us) server_negotiation.cc:407] Connection header received
0923 20:58:23.051547 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.051556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.051842 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.052334 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:23.053077 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.055019 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.058013 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.059538 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.059684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.059889 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.060166 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.060336 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10202,"thread_start_us":10619,"threads_started":1}
I20260923 20:58:23.060672 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.039366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33293 (local address 127.0.0.1:54582)
0923 20:58:23.051057 (+ 11691us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:23.051083 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:23.051329 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:23.052462 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:23.052467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:23.052493 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:23.052875 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.052886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.055196 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.055201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.057766 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.057784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.059764 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.059780 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.059947 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.059982 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:58:23.060091 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:58:23.060249 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11561,"mutex_wait_us":138,"thread_start_us":16072,"threads_started":1}
I20260923 20:58:23.061296 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.040320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.1:40439)
0923 20:58:23.050281 (+ 9961us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.050289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.052589 (+ 2300us) server_negotiation.cc:407] Connection header received
0923 20:58:23.052703 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.052713 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.052983 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.053179 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:23.053894 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.055487 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.058015 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.059538 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.059684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.059921 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.060773 (+ 852us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.060960 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9774,"thread_start_us":9974,"threads_started":1}
I20260923 20:58:23.061743 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.039873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40439 (local address 127.0.0.1:50032)
0923 20:58:23.052168 (+ 12295us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:23.052193 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:23.052322 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:23.053446 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:23.053450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:23.053476 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:23.053693 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.053703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.055686 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.055691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.057764 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.057800 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.060028 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.060033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.060267 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.060304 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:58:23.060401 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:58:23.060549 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12163,"mutex_wait_us":44,"thread_start_us":10894,"threads_started":1}
I20260923 20:58:23.065230 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea2c0aef524c71bf00997946bcafd4 (DEFAULT_TABLE table=test-table [id=90d85d05bdf04e4ea5adeb3c38d69414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:23.065264 17711 tablet_service.cc:1467] Processing CreateTablet for tablet 61ea2c0aef524c71bf00997946bcafd4 (DEFAULT_TABLE table=test-table [id=90d85d05bdf04e4ea5adeb3c38d69414]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:23.066835 17711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea2c0aef524c71bf00997946bcafd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:23.066843 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ea2c0aef524c71bf00997946bcafd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:23.072243 18063 tablet_bootstrap.cc:492] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Bootstrap starting.
I20260923 20:58:23.076397 18063 tablet_bootstrap.cc:654] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:23.077545 18063 log.cc:824] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:23.084144 18066 tablet_bootstrap.cc:492] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Bootstrap starting.
I20260923 20:58:23.086364 18063 tablet_bootstrap.cc:492] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: No bootstrap required, opened a new log
I20260923 20:58:23.086630 18065 tablet_bootstrap.cc:492] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Bootstrap starting.
I20260923 20:58:23.086740 18063 ts_tablet_manager.cc:1397] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260923 20:58:23.088411 18066 tablet_bootstrap.cc:654] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:23.089862 18066 log.cc:824] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:23.090137 18065 tablet_bootstrap.cc:654] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:23.091535 18065 log.cc:824] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:23.093580 18066 tablet_bootstrap.cc:492] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: No bootstrap required, opened a new log
I20260923 20:58:23.093935 18066 ts_tablet_manager.cc:1397] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260923 20:58:23.095057 18065 tablet_bootstrap.cc:492] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: No bootstrap required, opened a new log
I20260923 20:58:23.095348 18065 ts_tablet_manager.cc:1397] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:58:23.097481 18063 raft_consensus.cc:348] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.098048 18063 raft_consensus.cc:374] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:23.098183 18063 raft_consensus.cc:729] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d564e5c552fc45f9a49db782af8a8c7f, State: Initialized, Role: FOLLOWER
I20260923 20:58:23.098901 18063 consensus_queue.cc:260] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.105791 18044 heartbeater.cc:503] Master 127.0.97.62:44203 was elected leader, sending a full tablet report...
I20260923 20:58:23.106451 18063 ts_tablet_manager.cc:1428] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260923 20:58:23.106352 18065 raft_consensus.cc:348] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.106858 18065 raft_consensus.cc:374] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:23.105439 18066 raft_consensus.cc:348] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.106968 18065 raft_consensus.cc:729] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29abe1996b164974b34bf2c7336c1b34, State: Initialized, Role: FOLLOWER
I20260923 20:58:23.107095 18066 raft_consensus.cc:374] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:23.107215 18066 raft_consensus.cc:729] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70370753c9bb413e8051a370cc94ca99, State: Initialized, Role: FOLLOWER
I20260923 20:58:23.107648 18065 consensus_queue.cc:260] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.107854 18066 consensus_queue.cc:260] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.109637 18065 ts_tablet_manager.cc:1428] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 20:58:23.111521 17910 heartbeater.cc:503] Master 127.0.97.62:44203 was elected leader, sending a full tablet report...
I20260923 20:58:23.112317 18066 ts_tablet_manager.cc:1428] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
W20260923 20:58:23.120147 17777 tablet.cc:2369] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:23.149024 18070 raft_consensus.cc:482] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:58:23.149396 18070 raft_consensus.cc:504] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.152070 18070 leader_election.cc:288] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d564e5c552fc45f9a49db782af8a8c7f (127.0.97.3:46069), 70370753c9bb413e8051a370cc94ca99 (127.0.97.2:33293)
I20260923 20:58:23.160892 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.151727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42265 (local address 127.0.97.3:46069)
0923 20:58:23.151939 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.151946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.152904 (+ 958us) server_negotiation.cc:407] Connection header received
0923 20:58:23.153157 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.153161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.153259 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.153359 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:23.154138 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.155331 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.158569 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.159845 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.160001 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.160110 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.160465 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.160628 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:58:23.161000 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.151622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46069 (local address 127.0.97.1:42265)
0923 20:58:23.152754 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:23.152776 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:23.152987 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:23.153409 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:23.153412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:23.153438 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:23.153962 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.153975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.155494 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.155500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.158361 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.158382 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.160120 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.160126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.160284 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.160311 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 20:58:23.160403 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:58:23.160512 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":237,"spinlock_wait_cycles":39168,"thread_start_us":799,"threads_started":1}
I20260923 20:58:23.161140 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.151739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33293 (local address 127.0.97.1:58331)
0923 20:58:23.153125 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:23.153141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:23.153259 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:23.153585 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:23.153591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:23.153622 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:23.154076 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.154089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.155494 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.155500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.158361 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.158382 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.160276 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.160290 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.160460 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.160494 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:58:23.160590 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 20:58:23.160726 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":195,"thread_start_us":1091,"threads_started":1}
I20260923 20:58:23.161420 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.152153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58331 (local address 127.0.97.2:33293)
0923 20:58:23.152417 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.152426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.153248 (+ 822us) server_negotiation.cc:407] Connection header received
0923 20:58:23.153385 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.153389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.153460 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.153552 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:23.154260 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.155331 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.158568 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.159873 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.160000 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.160110 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.160660 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.160805 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:58:23.161970 18071 raft_consensus.cc:482] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:58:23.162343 18071 raft_consensus.cc:504] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.162444 17999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea2c0aef524c71bf00997946bcafd4" candidate_uuid: "29abe1996b164974b34bf2c7336c1b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d564e5c552fc45f9a49db782af8a8c7f" is_pre_election: true
I20260923 20:58:23.162452 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea2c0aef524c71bf00997946bcafd4" candidate_uuid: "29abe1996b164974b34bf2c7336c1b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70370753c9bb413e8051a370cc94ca99" is_pre_election: true
I20260923 20:58:23.163014 17999 raft_consensus.cc:2454] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29abe1996b164974b34bf2c7336c1b34 in term 0.
I20260923 20:58:23.163239 17865 raft_consensus.cc:2454] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29abe1996b164974b34bf2c7336c1b34 in term 0.
W20260923 20:58:23.163393 17911 tablet.cc:2369] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:23.163844 17664 leader_election.cc:302] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29abe1996b164974b34bf2c7336c1b34, d564e5c552fc45f9a49db782af8a8c7f; no voters:
I20260923 20:58:23.164353 18070 raft_consensus.cc:2781] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:58:23.164527 18070 raft_consensus.cc:482] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:58:23.164546 18071 leader_election.cc:288] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d564e5c552fc45f9a49db782af8a8c7f (127.0.97.3:46069), 29abe1996b164974b34bf2c7336c1b34 (127.0.97.1:40439)
I20260923 20:58:23.164646 18070 raft_consensus.cc:3037] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:23.168011 18070 raft_consensus.cc:504] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.169269 17999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea2c0aef524c71bf00997946bcafd4" candidate_uuid: "29abe1996b164974b34bf2c7336c1b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d564e5c552fc45f9a49db782af8a8c7f"
I20260923 20:58:23.169543 17999 raft_consensus.cc:3037] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:23.169545 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea2c0aef524c71bf00997946bcafd4" candidate_uuid: "29abe1996b164974b34bf2c7336c1b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70370753c9bb413e8051a370cc94ca99"
I20260923 20:58:23.169785 17865 raft_consensus.cc:3037] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:23.173316 17865 raft_consensus.cc:2454] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29abe1996b164974b34bf2c7336c1b34 in term 1.
I20260923 20:58:23.173308 17999 raft_consensus.cc:2454] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29abe1996b164974b34bf2c7336c1b34 in term 1.
I20260923 20:58:23.173792 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.165015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38465 (local address 127.0.97.3:46069)
0923 20:58:23.165213 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.165221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.165462 (+ 241us) server_negotiation.cc:407] Connection header received
0923 20:58:23.165777 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.165794 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.165910 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.165986 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:23.166807 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.167850 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.170500 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.171890 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.171998 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.172128 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.173339 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.173506 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:58:23.173863 17664 leader_election.cc:302] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29abe1996b164974b34bf2c7336c1b34, 70370753c9bb413e8051a370cc94ca99, d564e5c552fc45f9a49db782af8a8c7f; no voters:
I20260923 20:58:23.174029 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.164708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46069 (local address 127.0.97.2:38465)
0923 20:58:23.165342 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:23.165358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:23.165591 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:23.166214 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:23.166219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:23.166248 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:23.166656 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.166675 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.168000 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.168006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.170293 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.170310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.172360 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.172366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.172561 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.172599 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:58:23.172696 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:58:23.172870 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":322,"threads_started":1}
I20260923 20:58:23.174072 18070 leader_election.cc:288] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [CANDIDATE]: Term 1 election: Requested vote from peers d564e5c552fc45f9a49db782af8a8c7f (127.0.97.3:46069), 70370753c9bb413e8051a370cc94ca99 (127.0.97.2:33293)
I20260923 20:58:23.174291 18070 raft_consensus.cc:2781] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:23.174500 18070 raft_consensus.cc:686] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 1 LEADER]: Becoming Leader. State: Replica: 29abe1996b164974b34bf2c7336c1b34, State: Running, Role: LEADER
I20260923 20:58:23.174649 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.165606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52327 (local address 127.0.97.1:40439)
0923 20:58:23.165835 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.165841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.166484 (+ 643us) server_negotiation.cc:407] Connection header received
0923 20:58:23.166682 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.166687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.166761 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.166871 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:23.167486 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.168558 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.171416 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.172696 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.172805 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.172899 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.173622 (+ 723us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.173774 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:58:23.174738 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.165470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40439 (local address 127.0.97.2:52327)
0923 20:58:23.166358 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:23.166377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:23.166510 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:23.166895 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:23.166900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:23.166930 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:23.167302 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.167313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.168731 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.168737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.171179 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:23.171196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.173105 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.173111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:23.173325 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.173358 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:58:23.174236 (+ 878us) client_negotiation.cc:240] Negotiation successful
0923 20:58:23.174398 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":514,"threads_started":1}
I20260923 20:58:23.174935 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea2c0aef524c71bf00997946bcafd4" candidate_uuid: "70370753c9bb413e8051a370cc94ca99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d564e5c552fc45f9a49db782af8a8c7f" is_pre_election: true
I20260923 20:58:23.175130 18070 consensus_queue.cc:237] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } }
I20260923 20:58:23.175391 17998 raft_consensus.cc:2379] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 70370753c9bb413e8051a370cc94ca99 in current term 1: Already voted for candidate 29abe1996b164974b34bf2c7336c1b34 in this term.
I20260923 20:58:23.177205 17731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ea2c0aef524c71bf00997946bcafd4" candidate_uuid: "70370753c9bb413e8051a370cc94ca99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29abe1996b164974b34bf2c7336c1b34" is_pre_election: true
I20260923 20:58:23.178574 17798 leader_election.cc:302] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70370753c9bb413e8051a370cc94ca99; no voters: 29abe1996b164974b34bf2c7336c1b34, d564e5c552fc45f9a49db782af8a8c7f
I20260923 20:58:23.179061 18071 raft_consensus.cc:2726] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 20:58:23.180905 17586 catalog_manager.cc:5495] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29abe1996b164974b34bf2c7336c1b34 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "29abe1996b164974b34bf2c7336c1b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29abe1996b164974b34bf2c7336c1b34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40439 } health_report { overall_health: HEALTHY } } }
W20260923 20:58:23.212927 18045 tablet.cc:2369] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:23.221024 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.215224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.97.1:40439)
0923 20:58:23.215420 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.215425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.215545 (+ 120us) server_negotiation.cc:407] Connection header received
0923 20:58:23.215745 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.215748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.215832 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.215975 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:58:23.216348 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.217099 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.218772 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.219199 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.219307 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.219875 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:58:23.220418 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:58:23.220533 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.220703 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.220780 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:58:23.229372 17998 raft_consensus.cc:1261] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 1 FOLLOWER]: Refusing update from remote peer 29abe1996b164974b34bf2c7336c1b34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:23.229388 17865 raft_consensus.cc:1261] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 1 FOLLOWER]: Refusing update from remote peer 29abe1996b164974b34bf2c7336c1b34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:23.230345 18076 consensus_queue.cc:1035] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70370753c9bb413e8051a370cc94ca99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:23.230695 18070 consensus_queue.cc:1035] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d564e5c552fc45f9a49db782af8a8c7f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:23.260617 17586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:23.260928 17586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:23.264660 17586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37a14446f4ad4c3bb04df12c4627f6a3: Sending DeleteTablet for 3 replicas of tablet 61ea2c0aef524c71bf00997946bcafd4
I20260923 20:58:23.266041 17979 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea2c0aef524c71bf00997946bcafd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44280
I20260923 20:58:23.265990 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea2c0aef524c71bf00997946bcafd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54582
I20260923 20:58:23.266079 17711 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ea2c0aef524c71bf00997946bcafd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50032
I20260923 20:58:23.268743 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 17648
I20260923 20:58:23.278666 18090 tablet_replica.cc:330] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: stopping tablet replica
I20260923 20:58:23.279464 18090 raft_consensus.cc:2229] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:23.280473 18090 raft_consensus.cc:2258] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:23.283546 18090 ts_tablet_manager.cc:1905] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:23.290373 18089 tablet_replica.cc:330] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: stopping tablet replica
I20260923 20:58:23.291098 18089 raft_consensus.cc:2229] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:23.291388 18091 tablet_replica.cc:330] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: stopping tablet replica
I20260923 20:58:23.291823 18091 raft_consensus.cc:2229] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:58:23.291911 18089 raft_consensus.cc:2258] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:23.292805 18091 raft_consensus.cc:2258] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:23.294559 18089 ts_tablet_manager.cc:1905] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:23.295876 18090 ts_tablet_manager.cc:1918] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:23.296065 18090 log.cc:1196] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/61ea2c0aef524c71bf00997946bcafd4
I20260923 20:58:23.296641 18090 ts_tablet_manager.cc:1939] T 61ea2c0aef524c71bf00997946bcafd4 P d564e5c552fc45f9a49db782af8a8c7f: Deleting consensus metadata
I20260923 20:58:23.298468 18091 ts_tablet_manager.cc:1905] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:23.298689 17571 catalog_manager.cc:4841] TS d564e5c552fc45f9a49db782af8a8c7f (127.0.97.3:46069): tablet 61ea2c0aef524c71bf00997946bcafd4 (table test-table [id=90d85d05bdf04e4ea5adeb3c38d69414]) successfully deleted
I20260923 20:58:23.303313 18089 ts_tablet_manager.cc:1918] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:23.303627 18089 log.cc:1196] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/61ea2c0aef524c71bf00997946bcafd4
I20260923 20:58:23.304333 18089 ts_tablet_manager.cc:1939] T 61ea2c0aef524c71bf00997946bcafd4 P 70370753c9bb413e8051a370cc94ca99: Deleting consensus metadata
Sep 23 20:58:23 dist-test-slave-plcg krb5kdc[17542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:58:23.306586 17574 catalog_manager.cc:4841] TS 70370753c9bb413e8051a370cc94ca99 (127.0.97.2:33293): tablet 61ea2c0aef524c71bf00997946bcafd4 (table test-table [id=90d85d05bdf04e4ea5adeb3c38d69414]) successfully deleted
I20260923 20:58:23.307487 18091 ts_tablet_manager.cc:1918] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:23.307710 18091 log.cc:1196] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/61ea2c0aef524c71bf00997946bcafd4
I20260923 20:58:23.308287 18091 ts_tablet_manager.cc:1939] T 61ea2c0aef524c71bf00997946bcafd4 P 29abe1996b164974b34bf2c7336c1b34: Deleting consensus metadata
I20260923 20:58:23.310223 17571 catalog_manager.cc:4841] TS 29abe1996b164974b34bf2c7336c1b34 (127.0.97.1:40439): tablet 61ea2c0aef524c71bf00997946bcafd4 (table test-table [id=90d85d05bdf04e4ea5adeb3c38d69414]) successfully deleted
I20260923 20:58:23.313078 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.270617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.0.97.1:40439)
0923 20:58:23.271214 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.271220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.290414 (+ 19194us) server_negotiation.cc:407] Connection header received
0923 20:58:23.290653 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.290659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.290770 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.290892 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:23.292860 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.294073 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.299117 (+ 5044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.299848 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.299988 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.305879 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:23.305899 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:23.305921 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:23.305973 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:23.309799 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.310776 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.310793 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.310795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.310924 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.311338 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.311344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.311352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.311792 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:23.312158 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.312509 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.312759 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260923 20:58:23.313759 17771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:23.418855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 17648
I20260923 20:58:23.437959 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 17782
Sep 23 20:58:23 dist-test-slave-plcg krb5kdc[17542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:58:23.454026 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.438824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.97.2:33293)
0923 20:58:23.439027 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.439033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.439055 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:58:23.439140 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.439151 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.439233 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.439344 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:23.440566 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.441431 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.443037 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.443474 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.443618 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.448452 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:23.448468 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:23.448485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:23.448530 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:23.451374 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.452099 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.452107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.452109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.452241 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.452569 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.452572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.452574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.452947 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:23.453174 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.453496 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.453688 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:58:23.454713 17905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:23.548671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 17782
I20260923 20:58:23.566071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 17916
Sep 23 20:58:23 dist-test-slave-plcg krb5kdc[17542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:58:23.582404 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:23.566801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.0.97.3:46069)
0923 20:58:23.566980 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:23.566985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:23.567003 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:58:23.567152 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:23.567162 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:23.567226 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:23.567314 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:23.568663 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.569385 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.570901 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:23.571348 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:23.571459 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:23.576085 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:23.576110 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:23.576127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:23.576178 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:23.579240 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.580053 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.580061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.580064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.580219 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:23.580615 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:23.580622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:23.580626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:23.581034 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:23.581356 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:23.581671 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 20:58:23.582099 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:58:23.583181 18039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:23.673534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 17916
I20260923 20:58:23.690737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 17555
I20260923 20:58:23.691836 17617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:23.804540 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 17555
2026-09-23T20:58:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4456 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T20:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:23Z Disabled control of system clock
I20260923 20:58:23.889237 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34441
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:58:24.234715 18100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:24.235046 18100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:24.235186 18100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:24.242827 18100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:24.242959 18100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:24.243026 18100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:24.243110 18100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:24.243170 18100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:24.253067 18100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:24.254492 18100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:24.256386 18100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:24.268889 18105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:24.268831 18108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:24.268798 18106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:24.269765 18100 server_base.cc:1034] running on GCE node
I20260923 20:58:24.270674 18100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:24.272241 18100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:24.273460 18100 hybrid_clock.cc:648] HybridClock initialized: now 1790197104273385 us; error 68 us; skew 500 ppm
I20260923 20:58:24.273801 18100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:24.276260 18100 webserver.cc:460] Webserver started at http://127.0.97.62:39055/ using document root <none> and password file <none>
I20260923 20:58:24.277076 18100 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:24.277209 18100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:24.277576 18100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:24.280731 18100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "637383b86e9e46f2befbdea2bde8654a"
format_stamp: "Formatted at 2026-09-23 20:58:24 on dist-test-slave-plcg"
server_key: "3ee49b076bed9c4354319f8bc879fb6f"
server_key_iv: "c0ebb28f4f2226010968eafaebd28f1f"
server_key_version: "encryptionkey@0"
I20260923 20:58:24.281622 18100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "637383b86e9e46f2befbdea2bde8654a"
format_stamp: "Formatted at 2026-09-23 20:58:24 on dist-test-slave-plcg"
server_key: "3ee49b076bed9c4354319f8bc879fb6f"
server_key_iv: "c0ebb28f4f2226010968eafaebd28f1f"
server_key_version: "encryptionkey@0"
I20260923 20:58:24.287354 18100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:58:24.291328 18114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:24.292856 18100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:24.293075 18100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "637383b86e9e46f2befbdea2bde8654a"
format_stamp: "Formatted at 2026-09-23 20:58:24 on dist-test-slave-plcg"
server_key: "3ee49b076bed9c4354319f8bc879fb6f"
server_key_iv: "c0ebb28f4f2226010968eafaebd28f1f"
server_key_version: "encryptionkey@0"
I20260923 20:58:24.293320 18100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:24.307799 18100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:24.308696 18100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:24.309016 18100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:24.328557 18100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34441
I20260923 20:58:24.328575 18165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34441 every 8 connection(s)
I20260923 20:58:24.330107 18100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:24.334128 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:24.335546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 18100
I20260923 20:58:24.335740 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:24.336344 388 external_mini_cluster.cc:1428] Setting key 14ceb12d41c7b6697e1bb5a1e253d145
I20260923 20:58:24.348151 18166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a: Bootstrap starting.
I20260923 20:58:24.350778 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:24.338571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.97.62:34441)
0923 20:58:24.339664 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:24.339676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:24.339725 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:58:24.341374 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:24.341399 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:24.342033 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:24.342472 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:24.343071 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:24.344399 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:24.345886 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:24.346627 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:24.346857 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:24.348110 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:24.348118 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:24.348137 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:24.348793 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:24.348880 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:24.348904 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:24.349040 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:24.349281 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 20:58:24.349620 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":373,"threads_started":1}
I20260923 20:58:24.352377 18166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:24.353799 18166 log.cc:824] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:24.356925 18166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a: No bootstrap required, opened a new log
I20260923 20:58:24.365278 18166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637383b86e9e46f2befbdea2bde8654a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34441 } }
I20260923 20:58:24.365860 18166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:24.365983 18166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637383b86e9e46f2befbdea2bde8654a, State: Initialized, Role: FOLLOWER
I20260923 20:58:24.366727 18166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637383b86e9e46f2befbdea2bde8654a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34441 } }
I20260923 20:58:24.366984 18166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:24.367120 18166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:24.367323 18166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:24.369599 18166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637383b86e9e46f2befbdea2bde8654a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34441 } }
I20260923 20:58:24.370262 18166 leader_election.cc:302] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 637383b86e9e46f2befbdea2bde8654a; no voters:
I20260923 20:58:24.371038 18166 leader_election.cc:288] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:24.371222 18171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:24.371933 18171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [term 1 LEADER]: Becoming Leader. State: Replica: 637383b86e9e46f2befbdea2bde8654a, State: Running, Role: LEADER
I20260923 20:58:24.372438 18171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637383b86e9e46f2befbdea2bde8654a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34441 } }
I20260923 20:58:24.373229 18166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:24.376027 18172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "637383b86e9e46f2befbdea2bde8654a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637383b86e9e46f2befbdea2bde8654a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34441 } } }
I20260923 20:58:24.375764 18173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 637383b86e9e46f2befbdea2bde8654a. Latest consensus state: current_term: 1 leader_uuid: "637383b86e9e46f2befbdea2bde8654a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637383b86e9e46f2befbdea2bde8654a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34441 } } }
I20260923 20:58:24.376431 18172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:24.376473 18173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:24.387573 18180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:24.392333 18180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:24.401765 18180 catalog_manager.cc:1347] Generated new cluster ID: 86d9be8fef03496387424ce74e0d2f51
I20260923 20:58:24.401952 18180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:24.418447 18180 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:24.420450 18180 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:24.434098 18180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a: Generated new TSK 0
I20260923 20:58:24.435118 18180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:24.450687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34441
--builtin_ntp_servers=127.0.97.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:58:24.808650 18190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:24.808997 18190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:24.809092 18190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:24.816916 18190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:24.817030 18190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:24.817143 18190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:58:24.826787 18190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:24.828107 18190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:24.830044 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:24.843763 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:24.843804 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:24.843716 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:24.844972 18190 server_base.cc:1034] running on GCE node
I20260923 20:58:24.845750 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:24.846900 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:24.848157 18190 hybrid_clock.cc:648] HybridClock initialized: now 1790197104848119 us; error 68 us; skew 500 ppm
I20260923 20:58:24.848529 18190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:24.851258 18190 webserver.cc:460] Webserver started at http://127.0.97.1:46301/ using document root <none> and password file <none>
I20260923 20:58:24.852012 18190 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:24.852152 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:24.852489 18190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:24.855552 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "98ce61c403c740f68401f18f80002ab1"
format_stamp: "Formatted at 2026-09-23 20:58:24 on dist-test-slave-plcg"
server_key: "56408750766392f1a850f0b57bf8b3f7"
server_key_iv: "d4ac8c242bed6ab72d4a50987b4fb399"
server_key_version: "encryptionkey@0"
I20260923 20:58:24.856431 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "98ce61c403c740f68401f18f80002ab1"
format_stamp: "Formatted at 2026-09-23 20:58:24 on dist-test-slave-plcg"
server_key: "56408750766392f1a850f0b57bf8b3f7"
server_key_iv: "d4ac8c242bed6ab72d4a50987b4fb399"
server_key_version: "encryptionkey@0"
I20260923 20:58:24.862028 18190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:58:24.865998 18204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:24.867468 18190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:24.867691 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "98ce61c403c740f68401f18f80002ab1"
format_stamp: "Formatted at 2026-09-23 20:58:24 on dist-test-slave-plcg"
server_key: "56408750766392f1a850f0b57bf8b3f7"
server_key_iv: "d4ac8c242bed6ab72d4a50987b4fb399"
server_key_version: "encryptionkey@0"
I20260923 20:58:24.867961 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:24.884467 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:24.885366 18190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:24.885674 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:24.886758 18190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:24.888502 18190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:24.888646 18190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:24.888777 18190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:24.888865 18190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:24.923208 18190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43747
I20260923 20:58:24.923259 18316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43747 every 8 connection(s)
I20260923 20:58:24.924613 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:58:24.927266 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 18190
I20260923 20:58:24.927485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:58:24.928009 388 external_mini_cluster.cc:1428] Setting key 7c6aad7a5c49b8db827ada9f51d299dd
I20260923 20:58:24.930895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34441
--builtin_ntp_servers=127.0.97.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:58:24.939065 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:24.927297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50519 (local address 127.0.97.62:34441)
0923 20:58:24.927870 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:24.927876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:24.929291 (+ 1415us) server_negotiation.cc:407] Connection header received
0923 20:58:24.931257 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:24.931265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:24.931360 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:24.931481 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:24.932585 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:24.933791 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:24.935579 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:24.936359 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:24.936522 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:24.938051 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:24.938060 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:24.938067 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:24.938095 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:24.938149 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:24.938169 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:24.938269 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:24.938492 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 20:58:24.938618 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":296,"threads_started":1}
I20260923 20:58:24.940442 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:24.927946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34441 (local address 127.0.97.1:50519)
0923 20:58:24.929108 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:24.929164 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:24.930923 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:24.931710 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:24.931719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:24.931813 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:24.932400 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:24.932416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:24.933967 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:24.933971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:24.935402 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:24.935416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:24.936640 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:24.936646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:24.936873 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:24.937643 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:24.937646 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:24.937687 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:24.937774 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:24.938272 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:24.938280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:24.938286 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:58:24.938439 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 20:58:24.938957 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":416,"threads_started":1}
I20260923 20:58:24.942330 18317 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34441
I20260923 20:58:24.942690 18317 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:24.943547 18317 heartbeater.cc:511] Master 127.0.97.62:34441 requested a full tablet report, sending...
I20260923 20:58:24.945868 18131 ts_manager.cc:194] Registered new tserver with Master: 98ce61c403c740f68401f18f80002ab1 (127.0.97.1:43747)
I20260923 20:58:24.947409 18131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50519
W20260923 20:58:25.273949 18321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:25.274260 18321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:25.274459 18321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:25.282215 18321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:25.282330 18321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:25.282503 18321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:58:25.292006 18321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:25.293375 18321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:25.295375 18321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:25.309835 18326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:25.310040 18327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:25.309764 18329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:25.311008 18321 server_base.cc:1034] running on GCE node
I20260923 20:58:25.311719 18321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:25.312789 18321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:25.314040 18321 hybrid_clock.cc:648] HybridClock initialized: now 1790197105313995 us; error 64 us; skew 500 ppm
I20260923 20:58:25.314401 18321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:25.316903 18321 webserver.cc:460] Webserver started at http://127.0.97.2:42825/ using document root <none> and password file <none>
I20260923 20:58:25.317713 18321 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:25.317881 18321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:25.318240 18321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:25.321296 18321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8"
format_stamp: "Formatted at 2026-09-23 20:58:25 on dist-test-slave-plcg"
server_key: "3ce83e0a04371fe9bfb010fcaf01c789"
server_key_iv: "4188ded8956b275a78cb5830e560cfe4"
server_key_version: "encryptionkey@0"
I20260923 20:58:25.322253 18321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8"
format_stamp: "Formatted at 2026-09-23 20:58:25 on dist-test-slave-plcg"
server_key: "3ce83e0a04371fe9bfb010fcaf01c789"
server_key_iv: "4188ded8956b275a78cb5830e560cfe4"
server_key_version: "encryptionkey@0"
I20260923 20:58:25.327828 18321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260923 20:58:25.333258 18335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:25.335275 18321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:58:25.335484 18321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8"
format_stamp: "Formatted at 2026-09-23 20:58:25 on dist-test-slave-plcg"
server_key: "3ce83e0a04371fe9bfb010fcaf01c789"
server_key_iv: "4188ded8956b275a78cb5830e560cfe4"
server_key_version: "encryptionkey@0"
I20260923 20:58:25.335747 18321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:25.357501 18321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:25.358525 18321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:25.358841 18321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:25.359900 18321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:25.361487 18321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:25.361619 18321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:25.361758 18321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:25.361864 18321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:25.396911 18321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42613
I20260923 20:58:25.397002 18447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42613 every 8 connection(s)
I20260923 20:58:25.398371 18321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:58:25.399072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 18321
I20260923 20:58:25.399266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:58:25.399910 388 external_mini_cluster.cc:1428] Setting key 16c214202e1d35c3959a3ad6852beda3
I20260923 20:58:25.403620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34441
--builtin_ntp_servers=127.0.97.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:58:25.414949 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.401444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56805 (local address 127.0.97.62:34441)
0923 20:58:25.401651 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:25.401658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:25.403671 (+ 2013us) server_negotiation.cc:407] Connection header received
0923 20:58:25.406742 (+ 3071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:25.406748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:25.406826 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:25.406918 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:25.408457 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.409191 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.411909 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.412369 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.412559 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.414171 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:25.414176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:25.414179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:25.414201 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:25.414237 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:25.414258 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:25.414335 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:25.414612 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 20:58:25.414708 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:58:25.416551 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.402188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34441 (local address 127.0.97.2:56805)
0923 20:58:25.403455 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:25.403519 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:25.406366 (+ 2847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:25.407215 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:25.407230 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:25.407374 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:25.408227 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.408269 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.409396 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.409405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.411706 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.411727 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.412656 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.412664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.412931 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.413738 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:25.413744 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:25.413803 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:25.413942 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:25.414363 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:25.414378 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:25.414389 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:58:25.414582 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 20:58:25.415063 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":381,"threads_started":1}
I20260923 20:58:25.418177 18448 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34441
I20260923 20:58:25.418517 18448 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:25.419343 18448 heartbeater.cc:511] Master 127.0.97.62:34441 requested a full tablet report, sending...
I20260923 20:58:25.421010 18131 ts_manager.cc:194] Registered new tserver with Master: ca3bd9e5eef0417e9f8d25d9d78581a8 (127.0.97.2:42613)
I20260923 20:58:25.421927 18131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:56805
W20260923 20:58:25.749763 18451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:25.750212 18451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:25.750373 18451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:25.758226 18451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:25.758380 18451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:25.758569 18451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:58:25.768159 18451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:25.769572 18451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:25.771549 18451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:25.785980 18457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:25.785959 18459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:25.786078 18456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:25.786778 18451 server_base.cc:1034] running on GCE node
I20260923 20:58:25.787480 18451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:25.788533 18451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:25.789863 18451 hybrid_clock.cc:648] HybridClock initialized: now 1790197105789768 us; error 48 us; skew 500 ppm
I20260923 20:58:25.790251 18451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:25.792870 18451 webserver.cc:460] Webserver started at http://127.0.97.3:44257/ using document root <none> and password file <none>
I20260923 20:58:25.793644 18451 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:25.793793 18451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:25.794214 18451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:25.797410 18451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "39ea48a66a944703b192117c43efee52"
format_stamp: "Formatted at 2026-09-23 20:58:25 on dist-test-slave-plcg"
server_key: "b77517e63d1523ec670792e72927093d"
server_key_iv: "ad8f49d2741c19f1d779e89ae3e19e84"
server_key_version: "encryptionkey@0"
I20260923 20:58:25.798401 18451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "39ea48a66a944703b192117c43efee52"
format_stamp: "Formatted at 2026-09-23 20:58:25 on dist-test-slave-plcg"
server_key: "b77517e63d1523ec670792e72927093d"
server_key_iv: "ad8f49d2741c19f1d779e89ae3e19e84"
server_key_version: "encryptionkey@0"
I20260923 20:58:25.804081 18451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 20:58:25.807946 18465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:25.809478 18451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:58:25.809715 18451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "39ea48a66a944703b192117c43efee52"
format_stamp: "Formatted at 2026-09-23 20:58:25 on dist-test-slave-plcg"
server_key: "b77517e63d1523ec670792e72927093d"
server_key_iv: "ad8f49d2741c19f1d779e89ae3e19e84"
server_key_version: "encryptionkey@0"
I20260923 20:58:25.810036 18451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:25.825177 18451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:25.826192 18451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:25.826516 18451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:25.827651 18451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:25.829340 18451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:25.829485 18451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:25.829639 18451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:25.829737 18451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:25.866874 18451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39235
I20260923 20:58:25.866928 18577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39235 every 8 connection(s)
I20260923 20:58:25.868270 18451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:58:25.873775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 18451
I20260923 20:58:25.873970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:58:25.874586 388 external_mini_cluster.cc:1428] Setting key 9d5f3dcc173f09c64d2db8cd030d2317
I20260923 20:58:25.882863 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.871088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55025 (local address 127.0.97.62:34441)
0923 20:58:25.871304 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:25.871310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:25.872896 (+ 1586us) server_negotiation.cc:407] Connection header received
0923 20:58:25.875383 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:25.875393 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:25.875477 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:25.875592 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:25.877106 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.878219 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.879956 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.880401 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.880507 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.881960 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:25.881965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:25.881968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:25.881989 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:25.882022 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:25.882035 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:25.882152 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:25.882410 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 20:58:25.882519 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:58:25.884588 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.871520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34441 (local address 127.0.97.3:55025)
0923 20:58:25.872557 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:25.872633 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:25.875023 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:25.875869 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:25.875881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:25.876015 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:25.876903 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.876933 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.878374 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.878378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.879791 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.879804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.880620 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.880627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.880847 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.881568 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:25.881571 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:25.881615 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:25.881699 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:25.882142 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:25.882154 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:25.882164 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:58:25.882401 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 20:58:25.882891 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":345,"threads_started":1}
I20260923 20:58:25.887132 18578 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34441
I20260923 20:58:25.887511 18578 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:25.888634 18578 heartbeater.cc:511] Master 127.0.97.62:34441 requested a full tablet report, sending...
I20260923 20:58:25.890678 18130 ts_manager.cc:194] Registered new tserver with Master: 39ea48a66a944703b192117c43efee52 (127.0.97.3:39235)
I20260923 20:58:25.891579 18130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:55025
I20260923 20:58:25.904001 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:58:25.915161 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.908736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.0.97.62:34441)
0923 20:58:25.908993 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:25.909003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:25.909479 (+ 476us) server_negotiation.cc:407] Connection header received
0923 20:58:25.909704 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:25.909710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:25.909803 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:25.909941 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:25.910428 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.911142 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.912597 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.913041 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.913195 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.914370 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:25.914375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:25.914379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:25.914398 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:25.914434 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:25.914446 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:25.914549 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:25.914698 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 20:58:25.914838 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:58:25.920444 18130 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:58:25.927682 18130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:58:25.950991 18317 heartbeater.cc:503] Master 127.0.97.62:34441 was elected leader, sending a full tablet report...
I20260923 20:58:25.954957 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.940885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39235 (local address 127.0.0.1:32934)
0923 20:58:25.942615 (+ 1730us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:25.942663 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:25.942860 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:25.944085 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:25.944097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:25.944139 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:25.944639 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.944656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.947204 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.947213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.950337 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.950363 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.953534 (+ 3171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.953545 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.953902 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.953983 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 20:58:25.954147 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 20:58:25.954445 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1482,"thread_start_us":1293,"threads_started":1}
I20260923 20:58:25.956187 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.941131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.0.97.3:39235)
0923 20:58:25.941956 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:25.941965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:25.943093 (+ 1128us) server_negotiation.cc:407] Connection header received
0923 20:58:25.943220 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:25.943243 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:25.943523 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:25.943726 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:25.944914 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.946970 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.950662 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.952725 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.952894 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.953275 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:25.955474 (+ 2199us) server_negotiation.cc:299] Negotiation successful
0923 20:58:25.955728 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":373,"threads_started":1}
I20260923 20:58:25.964262 18513 tablet_service.cc:1467] Processing CreateTablet for tablet 75f1df379ef24960b974cad983d8c642 (DEFAULT_TABLE table=test-table [id=0a88d4234a474827a41cb4c095529378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:25.966089 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f1df379ef24960b974cad983d8c642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:25.966985 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.942955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42613 (local address 127.0.0.1:56352)
0923 20:58:25.956770 (+ 13815us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:25.956797 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:25.956946 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:25.960624 (+ 3678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:25.960632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:25.960672 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:25.961021 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.961034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.962836 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.962843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.964078 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.964091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.966073 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.966080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.966288 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.966323 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:58:25.966433 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:58:25.966617 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13636,"mutex_wait_us":1,"thread_start_us":18029,"threads_started":1}
I20260923 20:58:25.967065 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.943024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.0.97.2:42613)
0923 20:58:25.959622 (+ 16598us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:25.959632 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:25.959664 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:58:25.959796 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:25.959808 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:25.960114 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:25.961280 (+ 1166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:25.961496 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.962669 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.964263 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.965875 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.966034 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.966296 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:25.966493 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 20:58:25.966672 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16426,"thread_start_us":17269,"threads_started":1}
I20260923 20:58:25.968076 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.943671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.97.1:43747)
0923 20:58:25.957841 (+ 14170us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:25.957855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:25.958617 (+ 762us) server_negotiation.cc:407] Connection header received
0923 20:58:25.958743 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:25.958752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:25.959047 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:25.959281 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:25.960167 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.961803 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.964356 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.965934 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.966060 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.966743 (+ 683us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:25.967530 (+ 787us) server_negotiation.cc:299] Negotiation successful
0923 20:58:25.967711 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13942,"thread_start_us":13790,"threads_started":1}
I20260923 20:58:25.968717 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:25.943066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43747 (local address 127.0.0.1:35358)
0923 20:58:25.958322 (+ 15256us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:25.958344 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:25.958472 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:25.959580 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:25.959584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:25.959609 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:25.959983 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.959996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.961959 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.961968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.964083 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:25.964101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:25.966442 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:25.966448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:25.966901 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:25.966935 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:58:25.967038 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:58:25.968388 (+ 1350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15122,"mutex_wait_us":233,"thread_start_us":14985,"threads_started":1}
I20260923 20:58:25.971292 18383 tablet_service.cc:1467] Processing CreateTablet for tablet 75f1df379ef24960b974cad983d8c642 (DEFAULT_TABLE table=test-table [id=0a88d4234a474827a41cb4c095529378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:25.972007 18252 tablet_service.cc:1467] Processing CreateTablet for tablet 75f1df379ef24960b974cad983d8c642 (DEFAULT_TABLE table=test-table [id=0a88d4234a474827a41cb4c095529378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:25.972952 18383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f1df379ef24960b974cad983d8c642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:25.973508 18252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f1df379ef24960b974cad983d8c642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:25.983422 18596 tablet_bootstrap.cc:492] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Bootstrap starting.
I20260923 20:58:25.987100 18597 tablet_bootstrap.cc:492] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Bootstrap starting.
I20260923 20:58:25.988060 18598 tablet_bootstrap.cc:492] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Bootstrap starting.
I20260923 20:58:25.988287 18596 tablet_bootstrap.cc:654] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:25.989794 18596 log.cc:824] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:25.991132 18597 tablet_bootstrap.cc:654] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:25.991660 18598 tablet_bootstrap.cc:654] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:25.992465 18597 log.cc:824] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:25.992751 18598 log.cc:824] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:25.993577 18596 tablet_bootstrap.cc:492] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: No bootstrap required, opened a new log
I20260923 20:58:25.993965 18596 ts_tablet_manager.cc:1397] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260923 20:58:25.996986 18598 tablet_bootstrap.cc:492] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: No bootstrap required, opened a new log
I20260923 20:58:25.997241 18597 tablet_bootstrap.cc:492] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: No bootstrap required, opened a new log
I20260923 20:58:25.997349 18598 ts_tablet_manager.cc:1397] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260923 20:58:25.997515 18597 ts_tablet_manager.cc:1397] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260923 20:58:26.004907 18598 raft_consensus.cc:348] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.005450 18598 raft_consensus.cc:374] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:26.005564 18598 raft_consensus.cc:729] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3bd9e5eef0417e9f8d25d9d78581a8, State: Initialized, Role: FOLLOWER
I20260923 20:58:26.005368 18596 raft_consensus.cc:348] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.005805 18596 raft_consensus.cc:374] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:26.005945 18596 raft_consensus.cc:729] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ea48a66a944703b192117c43efee52, State: Initialized, Role: FOLLOWER
I20260923 20:58:26.006680 18596 consensus_queue.cc:260] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.006680 18598 consensus_queue.cc:260] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.008527 18448 heartbeater.cc:503] Master 127.0.97.62:34441 was elected leader, sending a full tablet report...
I20260923 20:58:26.009117 18598 ts_tablet_manager.cc:1428] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260923 20:58:26.008877 18597 raft_consensus.cc:348] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.009346 18597 raft_consensus.cc:374] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:26.009454 18597 raft_consensus.cc:729] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ce61c403c740f68401f18f80002ab1, State: Initialized, Role: FOLLOWER
I20260923 20:58:26.010143 18597 consensus_queue.cc:260] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.011010 18578 heartbeater.cc:503] Master 127.0.97.62:34441 was elected leader, sending a full tablet report...
I20260923 20:58:26.014182 18596 ts_tablet_manager.cc:1428] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
I20260923 20:58:26.014936 18597 ts_tablet_manager.cc:1428] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260923 20:58:26.117206 18603 raft_consensus.cc:482] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:58:26.117568 18603 raft_consensus.cc:504] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.119369 18603 leader_election.cc:288] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39ea48a66a944703b192117c43efee52 (127.0.97.3:39235), 98ce61c403c740f68401f18f80002ab1 (127.0.97.1:43747)
W20260923 20:58:26.120085 18579 tablet.cc:2369] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:26.126608 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.119973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35443 (local address 127.0.97.3:39235)
0923 20:58:26.120204 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:26.120211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:26.120562 (+ 351us) server_negotiation.cc:407] Connection header received
0923 20:58:26.120946 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:26.120956 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:26.121077 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:26.121154 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:26.121919 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.122895 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.124816 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.125623 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.125778 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.125921 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:26.126239 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 20:58:26.126363 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 20:58:26.126680 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.119763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39235 (local address 127.0.97.2:35443)
0923 20:58:26.120454 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:26.120474 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:26.120715 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:26.121356 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:26.121362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:26.121395 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:26.121740 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:26.121748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.123054 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.123059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:26.124572 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:26.124594 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.125880 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.125887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:26.126060 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.126084 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:58:26.126199 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:58:26.126345 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":320,"threads_started":1}
I20260923 20:58:26.126856 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.120681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40619 (local address 127.0.97.1:43747)
0923 20:58:26.120875 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:26.120881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:26.121533 (+ 652us) server_negotiation.cc:407] Connection header received
0923 20:58:26.121786 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:26.121792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:26.121907 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:26.122006 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:26.122465 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.123339 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.125075 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.125945 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.126094 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.126192 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:26.126472 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:58:26.126600 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:58:26.127247 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.120530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43747 (local address 127.0.97.2:40619)
0923 20:58:26.121405 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:26.121420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:26.121629 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:26.122025 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:26.122028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:26.122047 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:26.122320 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:26.122327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.123487 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.123499 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:26.124880 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:26.124892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.126131 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.126137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:26.126315 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.126338 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:58:26.126445 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:58:26.126573 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":601,"threads_started":1}
I20260923 20:58:26.127871 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f1df379ef24960b974cad983d8c642" candidate_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ea48a66a944703b192117c43efee52" is_pre_election: true
I20260923 20:58:26.128229 18272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f1df379ef24960b974cad983d8c642" candidate_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98ce61c403c740f68401f18f80002ab1" is_pre_election: true
I20260923 20:58:26.128396 18533 raft_consensus.cc:2454] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3bd9e5eef0417e9f8d25d9d78581a8 in term 0.
I20260923 20:58:26.128701 18272 raft_consensus.cc:2454] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3bd9e5eef0417e9f8d25d9d78581a8 in term 0.
I20260923 20:58:26.129125 18339 leader_election.cc:302] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39ea48a66a944703b192117c43efee52, ca3bd9e5eef0417e9f8d25d9d78581a8; no voters:
I20260923 20:58:26.129637 18603 raft_consensus.cc:2781] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:58:26.129822 18603 raft_consensus.cc:482] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:58:26.130013 18603 raft_consensus.cc:3037] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:26.132483 18603 raft_consensus.cc:504] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.133160 18603 leader_election.cc:288] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [CANDIDATE]: Term 1 election: Requested vote from peers 39ea48a66a944703b192117c43efee52 (127.0.97.3:39235), 98ce61c403c740f68401f18f80002ab1 (127.0.97.1:43747)
I20260923 20:58:26.133750 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f1df379ef24960b974cad983d8c642" candidate_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ea48a66a944703b192117c43efee52"
I20260923 20:58:26.133790 18272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f1df379ef24960b974cad983d8c642" candidate_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98ce61c403c740f68401f18f80002ab1"
I20260923 20:58:26.134011 18533 raft_consensus.cc:3037] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:26.134128 18272 raft_consensus.cc:3037] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:26.136695 18533 raft_consensus.cc:2454] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3bd9e5eef0417e9f8d25d9d78581a8 in term 1.
I20260923 20:58:26.136904 18272 raft_consensus.cc:2454] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3bd9e5eef0417e9f8d25d9d78581a8 in term 1.
I20260923 20:58:26.137178 18339 leader_election.cc:302] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39ea48a66a944703b192117c43efee52, ca3bd9e5eef0417e9f8d25d9d78581a8; no voters:
I20260923 20:58:26.137620 18603 raft_consensus.cc:2781] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:26.138146 18603 raft_consensus.cc:686] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 1 LEADER]: Becoming Leader. State: Replica: ca3bd9e5eef0417e9f8d25d9d78581a8, State: Running, Role: LEADER
I20260923 20:58:26.138705 18603 consensus_queue.cc:237] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } }
I20260923 20:58:26.144073 18130 catalog_manager.cc:5495] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca3bd9e5eef0417e9f8d25d9d78581a8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca3bd9e5eef0417e9f8d25d9d78581a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42613 } health_report { overall_health: HEALTHY } } }
W20260923 20:58:26.150553 18449 tablet.cc:2369] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:58:26.177191 18318 tablet.cc:2369] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:26.193651 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.188127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.0.97.2:42613)
0923 20:58:26.188356 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:26.188364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:26.188386 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:58:26.188484 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:26.188488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:26.188582 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:26.188762 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:58:26.189136 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.189917 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.191356 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.191789 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.191931 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.192455 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:58:26.193019 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:58:26.193185 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:26.193289 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 20:58:26.193373 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:58:26.202829 18272 raft_consensus.cc:1261] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 1 FOLLOWER]: Refusing update from remote peer ca3bd9e5eef0417e9f8d25d9d78581a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:26.202862 18533 raft_consensus.cc:1261] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 1 FOLLOWER]: Refusing update from remote peer ca3bd9e5eef0417e9f8d25d9d78581a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:26.203891 18607 consensus_queue.cc:1035] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39ea48a66a944703b192117c43efee52" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:26.204335 18603 consensus_queue.cc:1035] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98ce61c403c740f68401f18f80002ab1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:26.218070 18610 mvcc.cc:205] Tried to move back new op lower bound from 7332647346996412416 to 7332647346748575744. Current Snapshot: MvccSnapshot[applied={T|T < 7332647346996412416}]
I20260923 20:58:26.230506 18612 mvcc.cc:205] Tried to move back new op lower bound from 7332647346996412416 to 7332647346748575744. Current Snapshot: MvccSnapshot[applied={T|T < 7332647346996412416}]
I20260923 20:58:26.243155 18130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:26.243431 18130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:26.247308 18130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 637383b86e9e46f2befbdea2bde8654a: Sending DeleteTablet for 3 replicas of tablet 75f1df379ef24960b974cad983d8c642
I20260923 20:58:26.248668 18252 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f1df379ef24960b974cad983d8c642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:26 UTC) from {username='slave'} at 127.0.0.1:35358
I20260923 20:58:26.248627 18513 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f1df379ef24960b974cad983d8c642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:26 UTC) from {username='slave'} at 127.0.0.1:32934
I20260923 20:58:26.248711 18383 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f1df379ef24960b974cad983d8c642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:26 UTC) from {username='slave'} at 127.0.0.1:56352
I20260923 20:58:26.250890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 18190
I20260923 20:58:26.261322 18621 tablet_replica.cc:330] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: stopping tablet replica
I20260923 20:58:26.261708 18622 tablet_replica.cc:330] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: stopping tablet replica
I20260923 20:58:26.262387 18622 raft_consensus.cc:2229] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:58:26.262939 18621 raft_consensus.cc:2229] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:26.263538 18622 raft_consensus.cc:2258] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:26.264137 18621 raft_consensus.cc:2258] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:26.264753 18620 tablet_replica.cc:330] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: stopping tablet replica
I20260923 20:58:26.265228 18620 raft_consensus.cc:2229] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:26.265775 18620 raft_consensus.cc:2258] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:26.266180 18622 ts_tablet_manager.cc:1905] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:26.266825 18621 ts_tablet_manager.cc:1905] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:26.268325 18620 ts_tablet_manager.cc:1905] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:26.274403 18621 ts_tablet_manager.cc:1918] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:26.274652 18621 log.cc:1196] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/75f1df379ef24960b974cad983d8c642
I20260923 20:58:26.275236 18621 ts_tablet_manager.cc:1939] T 75f1df379ef24960b974cad983d8c642 P 98ce61c403c740f68401f18f80002ab1: Deleting consensus metadata
I20260923 20:58:26.275856 18620 ts_tablet_manager.cc:1918] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:26.276067 18620 log.cc:1196] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/75f1df379ef24960b974cad983d8c642
I20260923 20:58:26.276654 18620 ts_tablet_manager.cc:1939] T 75f1df379ef24960b974cad983d8c642 P 39ea48a66a944703b192117c43efee52: Deleting consensus metadata
I20260923 20:58:26.277626 18118 catalog_manager.cc:4841] TS 98ce61c403c740f68401f18f80002ab1 (127.0.97.1:43747): tablet 75f1df379ef24960b974cad983d8c642 (table test-table [id=0a88d4234a474827a41cb4c095529378]) successfully deleted
I20260923 20:58:26.278642 18118 catalog_manager.cc:4841] TS 39ea48a66a944703b192117c43efee52 (127.0.97.3:39235): tablet 75f1df379ef24960b974cad983d8c642 (table test-table [id=0a88d4234a474827a41cb4c095529378]) successfully deleted
I20260923 20:58:26.280978 18622 ts_tablet_manager.cc:1918] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:26.281167 18622 log.cc:1196] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/75f1df379ef24960b974cad983d8c642
I20260923 20:58:26.281764 18622 ts_tablet_manager.cc:1939] T 75f1df379ef24960b974cad983d8c642 P ca3bd9e5eef0417e9f8d25d9d78581a8: Deleting consensus metadata
I20260923 20:58:26.283679 18116 catalog_manager.cc:4841] TS ca3bd9e5eef0417e9f8d25d9d78581a8 (127.0.97.2:42613): tablet 75f1df379ef24960b974cad983d8c642 (table test-table [id=0a88d4234a474827a41cb4c095529378]) successfully deleted
I20260923 20:58:26.284914 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.252432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.0.97.1:43747)
0923 20:58:26.252620 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:26.252626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:26.272035 (+ 19409us) server_negotiation.cc:407] Connection header received
0923 20:58:26.272340 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:26.272347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:26.272490 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:26.272582 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:26.273409 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.274678 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.281348 (+ 6670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.281800 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.282011 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.283244 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:26.283254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:26.283268 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:26.284081 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:26.284133 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:26.284155 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:26.284304 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:26.284446 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 20:58:26.284570 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:58:26.285725 18312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:26.392706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 18190
I20260923 20:58:26.411586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 18321
I20260923 20:58:26.419638 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.412367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.0.97.2:42613)
0923 20:58:26.412857 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:26.412861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:26.412878 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:58:26.412933 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:26.412937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:26.413008 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:26.413124 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:26.413534 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.414660 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.416322 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.416800 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.416950 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.418151 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:26.418160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:26.418174 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:26.418794 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:26.418848 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:26.418868 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:26.418990 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:26.419111 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 20:58:26.419231 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260923 20:58:26.420487 18443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:26.536384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 18321
I20260923 20:58:26.555892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 18451
I20260923 20:58:26.562911 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:26.556717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.97.3:39235)
0923 20:58:26.556897 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:26.556902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:26.556925 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:58:26.557096 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:26.557100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:26.557195 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:26.557303 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:26.557644 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.558491 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.559898 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:26.560321 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:26.560424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:26.561506 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:26.561515 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:26.561530 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:26.562252 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:26.562306 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:26.562328 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:26.562434 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:26.562534 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 20:58:26.562631 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:58:26.563616 18573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:26.653518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 18451
I20260923 20:58:26.670807 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 18100
I20260923 20:58:26.671916 18161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:26.785003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 18100
2026-09-23T20:58:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2977 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:26 dist-test-slave-plcg krb5kdc[18630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:26 dist-test-slave-plcg krb5kdc[18630](info): set up 2 sockets
Sep 23 20:58:26 dist-test-slave-plcg krb5kdc[18630](info): commencing operation
krb5kdc: starting...
W20260923 20:58:27.921917 388 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:28 dist-test-slave-plcg krb5kdc[18630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:58:28.182408 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41491
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:28.537458 18643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:28.537787 18643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:28.537961 18643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:28.545555 18643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:28.545701 18643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:28.545750 18643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:28.545805 18643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:28.545930 18643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:28.556313 18643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:28.557770 18643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:28.559242 18643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:28.572625 18648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:28.572602 18651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:28.572567 18649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:28.573529 18643 server_base.cc:1034] running on GCE node
I20260923 20:58:28.574504 18643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:28.576077 18643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:28.577325 18643 hybrid_clock.cc:648] HybridClock initialized: now 1790197108577282 us; error 69 us; skew 500 ppm
I20260923 20:58:28.577720 18643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:28 dist-test-slave-plcg krb5kdc[18630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:28.585206 18643 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:58:28.587077 18643 webserver.cc:460] Webserver started at http://127.0.97.62:35835/ using document root <none> and password file <none>
I20260923 20:58:28.587867 18643 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:28.588020 18643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:28.588349 18643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:28.591686 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "228127d6672e4841b5eed25d1f19c6d0"
format_stamp: "Formatted at 2026-09-23 20:58:28 on dist-test-slave-plcg"
server_key: "be852374f79a0410bc15899f76b744b0"
server_key_iv: "d589c1d388430c4c831e985f751fb46f"
server_key_version: "encryptionkey@0"
I20260923 20:58:28.592543 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "228127d6672e4841b5eed25d1f19c6d0"
format_stamp: "Formatted at 2026-09-23 20:58:28 on dist-test-slave-plcg"
server_key: "be852374f79a0410bc15899f76b744b0"
server_key_iv: "d589c1d388430c4c831e985f751fb46f"
server_key_version: "encryptionkey@0"
I20260923 20:58:28.598373 18643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 20:58:28.602272 18658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:28.603797 18643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:28.603955 18643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "228127d6672e4841b5eed25d1f19c6d0"
format_stamp: "Formatted at 2026-09-23 20:58:28 on dist-test-slave-plcg"
server_key: "be852374f79a0410bc15899f76b744b0"
server_key_iv: "d589c1d388430c4c831e985f751fb46f"
server_key_version: "encryptionkey@0"
I20260923 20:58:28.604135 18643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:28.622737 18643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:28.627490 18643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:28.627813 18643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:28.647630 18643 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41491
I20260923 20:58:28.647686 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41491 every 8 connection(s)
I20260923 20:58:28.649187 18643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:28.653316 18710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:28.658500 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 18643
I20260923 20:58:28.658718 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:28.659147 388 external_mini_cluster.cc:1428] Setting key 94af095eddb02e3a963fa3b55c9d6e9a
I20260923 20:58:28.666090 18710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0: Bootstrap starting.
I20260923 20:58:28.670414 18710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:28.671792 18710 log.cc:824] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:58:28 dist-test-slave-plcg krb5kdc[18630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:28.675750 18710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0: No bootstrap required, opened a new log
I20260923 20:58:28.685257 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:28.661174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.0.97.62:41491)
0923 20:58:28.662137 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:28.662151 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:28.662212 (+ 61us) server_negotiation.cc:407] Connection header received
0923 20:58:28.664159 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:28.664196 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:28.664645 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:28.665034 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:28.667261 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:28.668642 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:28.670793 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:28.671545 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:28.671694 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:28.676954 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:28.676962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:28.676979 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:28.677032 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:28.680436 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:28.681300 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:28.681308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:28.681315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:28.681513 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:28.682040 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:28.682045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:28.682047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:28.682725 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:28.683113 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:28.683611 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 20:58:28.684100 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":417,"threads_started":1}
I20260923 20:58:28.685179 18710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228127d6672e4841b5eed25d1f19c6d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41491 } }
I20260923 20:58:28.685729 18710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:28.685921 18710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228127d6672e4841b5eed25d1f19c6d0, State: Initialized, Role: FOLLOWER
I20260923 20:58:28.686676 18710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228127d6672e4841b5eed25d1f19c6d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41491 } }
I20260923 20:58:28.686977 18710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:28.687155 18710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:28.687377 18710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:28.690457 18710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228127d6672e4841b5eed25d1f19c6d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41491 } }
I20260923 20:58:28.691167 18710 leader_election.cc:302] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 228127d6672e4841b5eed25d1f19c6d0; no voters:
I20260923 20:58:28.691875 18710 leader_election.cc:288] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:28.691987 18715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:28.692951 18715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [term 1 LEADER]: Becoming Leader. State: Replica: 228127d6672e4841b5eed25d1f19c6d0, State: Running, Role: LEADER
I20260923 20:58:28.693549 18715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228127d6672e4841b5eed25d1f19c6d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41491 } }
I20260923 20:58:28.694377 18710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:28.697219 18716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "228127d6672e4841b5eed25d1f19c6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228127d6672e4841b5eed25d1f19c6d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41491 } } }
I20260923 20:58:28.698042 18716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:28.697448 18717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 228127d6672e4841b5eed25d1f19c6d0. Latest consensus state: current_term: 1 leader_uuid: "228127d6672e4841b5eed25d1f19c6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228127d6672e4841b5eed25d1f19c6d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41491 } } }
I20260923 20:58:28.698793 18717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:28.703648 18724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:28.709532 18724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:28.719406 18724 catalog_manager.cc:1347] Generated new cluster ID: e1a8a6f5a80b463d9f4b23db0fc50fe6
I20260923 20:58:28.719538 18724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:28.731287 18724 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:28.733285 18724 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:28.746498 18724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0: Generated new TSK 0
I20260923 20:58:28.747740 18724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:58:28.821578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41491
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:29.178148 18736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:29.178555 18736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:29.178727 18736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:29.186290 18736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:29.186462 18736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:29.186667 18736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:58:29.197135 18736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:29.198601 18736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:29.200070 18736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:29.214608 18741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:29.214529 18744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:29.214560 18742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:29.215421 18736 server_base.cc:1034] running on GCE node
I20260923 20:58:29.216128 18736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:29.217172 18736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:29.218417 18736 hybrid_clock.cc:648] HybridClock initialized: now 1790197109218347 us; error 67 us; skew 500 ppm
I20260923 20:58:29.218777 18736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:29 dist-test-slave-plcg krb5kdc[18630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:29.222128 18736 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:58:29.223898 18736 webserver.cc:460] Webserver started at http://127.0.97.1:40687/ using document root <none> and password file <none>
I20260923 20:58:29.224706 18736 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:29.224870 18736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:29.225210 18736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:29.228406 18736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "904c82fa81eb40ccb4dac0057dec7b4a"
format_stamp: "Formatted at 2026-09-23 20:58:29 on dist-test-slave-plcg"
server_key: "87e8bf896169a1ad76ac387fdda8b85f"
server_key_iv: "b970b8d74acdc5c433090db69a05d285"
server_key_version: "encryptionkey@0"
I20260923 20:58:29.229437 18736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "904c82fa81eb40ccb4dac0057dec7b4a"
format_stamp: "Formatted at 2026-09-23 20:58:29 on dist-test-slave-plcg"
server_key: "87e8bf896169a1ad76ac387fdda8b85f"
server_key_iv: "b970b8d74acdc5c433090db69a05d285"
server_key_version: "encryptionkey@0"
I20260923 20:58:29.235278 18736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:58:29.239200 18751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:29.240623 18736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:29.240823 18736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "904c82fa81eb40ccb4dac0057dec7b4a"
format_stamp: "Formatted at 2026-09-23 20:58:29 on dist-test-slave-plcg"
server_key: "87e8bf896169a1ad76ac387fdda8b85f"
server_key_iv: "b970b8d74acdc5c433090db69a05d285"
server_key_version: "encryptionkey@0"
I20260923 20:58:29.241052 18736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:29.264508 18736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:29.268994 18736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:29.269323 18736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:29.270499 18736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:29.271917 18736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:29.272028 18736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:29.272176 18736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:29.272249 18736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:29.306645 18736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33919
I20260923 20:58:29.306742 18863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33919 every 8 connection(s)
I20260923 20:58:29.308104 18736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:58:29.310913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 18736
I20260923 20:58:29.311107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:58:29.311627 388 external_mini_cluster.cc:1428] Setting key adc295a34b438b875c861255f7829275
Sep 23 20:58:29 dist-test-slave-plcg krb5kdc[18630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:29.331637 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:29.311093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56797 (local address 127.0.97.62:41491)
0923 20:58:29.311607 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:29.311611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:29.312887 (+ 1276us) server_negotiation.cc:407] Connection header received
0923 20:58:29.315215 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:29.315224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:29.315320 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:29.315444 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:29.317132 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.318341 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.320006 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.320775 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.320932 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:29.325397 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:29.325401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:29.325405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:29.325441 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:29.328091 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:29.329114 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:29.329121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:29.329124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:29.329267 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:29.329785 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:29.329788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:29.329790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:29.330112 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:29.330290 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:29.331187 (+ 897us) server_negotiation.cc:299] Negotiation successful
0923 20:58:29.331372 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260923 20:58:29.333235 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:29.311775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41491 (local address 127.0.97.1:56797)
0923 20:58:29.312713 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:29.312754 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:29.314882 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:29.315701 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:29.315710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:29.316310 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:29.316928 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:29.316946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.318532 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.318537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:29.319837 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:29.319850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.320997 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.321002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:29.321218 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:29.322001 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:29.322025 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:29.325128 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:29.328253 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:29.328266 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:29.328287 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:29.328949 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:29.329389 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:29.329397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:29.329403 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:29.329676 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:29.330256 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:29.330267 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:29.330777 (+ 510us) client_negotiation.cc:769] Sending connection context
0923 20:58:29.331097 (+ 320us) client_negotiation.cc:240] Negotiation successful
0923 20:58:29.331717 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":365,"threads_started":1}
I20260923 20:58:29.334988 18864 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41491
I20260923 20:58:29.335326 18864 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:29.336079 18864 heartbeater.cc:511] Master 127.0.97.62:41491 requested a full tablet report, sending...
I20260923 20:58:29.338413 18675 ts_manager.cc:194] Registered new tserver with Master: 904c82fa81eb40ccb4dac0057dec7b4a (127.0.97.1:33919)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:58:29.339984 18675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:58:29.373134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41491
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:29.719708 18870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:29.720012 18870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:29.720109 18870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:29.727663 18870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:29.727789 18870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:29.727910 18870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:58:29.737890 18870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:29.739194 18870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:29.740525 18870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:29.754467 18876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:29.754407 18875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:29.754500 18878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:29.755336 18870 server_base.cc:1034] running on GCE node
I20260923 20:58:29.755975 18870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:29.757061 18870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:29.758280 18870 hybrid_clock.cc:648] HybridClock initialized: now 1790197109758231 us; error 44 us; skew 500 ppm
I20260923 20:58:29.758574 18870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:29 dist-test-slave-plcg krb5kdc[18630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:29.761895 18870 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:58:29.763468 18870 webserver.cc:460] Webserver started at http://127.0.97.2:42879/ using document root <none> and password file <none>
I20260923 20:58:29.764199 18870 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:29.764335 18870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:29.764685 18870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:29.767799 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "5c82eeddf4a64299947f973743824a79"
format_stamp: "Formatted at 2026-09-23 20:58:29 on dist-test-slave-plcg"
server_key: "70d3213d373f0d746ecf4f131943b9f3"
server_key_iv: "b8d0c91c82617be3cab6056431a3f28d"
server_key_version: "encryptionkey@0"
I20260923 20:58:29.768802 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5c82eeddf4a64299947f973743824a79"
format_stamp: "Formatted at 2026-09-23 20:58:29 on dist-test-slave-plcg"
server_key: "70d3213d373f0d746ecf4f131943b9f3"
server_key_iv: "b8d0c91c82617be3cab6056431a3f28d"
server_key_version: "encryptionkey@0"
I20260923 20:58:29.774524 18870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:58:29.778455 18885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:29.780082 18870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:58:29.780314 18870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "5c82eeddf4a64299947f973743824a79"
format_stamp: "Formatted at 2026-09-23 20:58:29 on dist-test-slave-plcg"
server_key: "70d3213d373f0d746ecf4f131943b9f3"
server_key_iv: "b8d0c91c82617be3cab6056431a3f28d"
server_key_version: "encryptionkey@0"
I20260923 20:58:29.780565 18870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:29.803030 18870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:29.808470 18870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:29.808763 18870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:29.810177 18870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:29.811672 18870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:29.811760 18870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:29.811853 18870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:29.811902 18870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:29.848126 18870 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43013
I20260923 20:58:29.848177 18997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43013 every 8 connection(s)
I20260923 20:58:29.849587 18870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:58:29.859014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 18870
I20260923 20:58:29.859201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:58:29.859644 388 external_mini_cluster.cc:1428] Setting key 5af90b171d15275e44e56539336993d9
Sep 23 20:58:29 dist-test-slave-plcg krb5kdc[18630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:29.873138 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:29.852447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37571 (local address 127.0.97.62:41491)
0923 20:58:29.852905 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:29.852909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:29.854054 (+ 1145us) server_negotiation.cc:407] Connection header received
0923 20:58:29.855975 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:29.855981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:29.856079 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:29.856227 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:29.857934 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.858747 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.860820 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.861294 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.861452 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:29.865722 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:29.865732 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:29.865740 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:29.865793 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:29.869604 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:29.870604 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:29.870609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:29.870611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:29.870756 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:29.871253 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:29.871257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:29.871259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:29.871540 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:29.871716 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:29.872480 (+ 764us) server_negotiation.cc:299] Negotiation successful
0923 20:58:29.872745 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260923 20:58:29.874296 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:29.853088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41491 (local address 127.0.97.2:37571)
0923 20:58:29.853886 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:29.853932 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:29.855670 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:29.856489 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:29.856497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:29.857069 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:29.857715 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:29.857734 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.858972 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.858976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:29.860662 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:29.860682 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:29.861599 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:29.861608 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:29.861965 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:29.862796 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:29.862826 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:29.865409 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:29.869810 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:29.869868 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:29.869893 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:29.870405 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:29.870886 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:29.870891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:29.870896 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:29.871109 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:29.871687 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:29.871695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:29.872086 (+ 391us) client_negotiation.cc:769] Sending connection context
0923 20:58:29.872375 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 20:58:29.872955 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":310,"threads_started":1}
I20260923 20:58:29.875910 18998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41491
I20260923 20:58:29.876278 18998 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:29.877259 18998 heartbeater.cc:511] Master 127.0.97.62:41491 requested a full tablet report, sending...
I20260923 20:58:29.879209 18675 ts_manager.cc:194] Registered new tserver with Master: 5c82eeddf4a64299947f973743824a79 (127.0.97.2:43013)
I20260923 20:58:29.880511 18675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37571
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:58:29.921440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41491
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:30.268070 19004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:30.268383 19004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:30.268555 19004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:30.276098 19004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:30.276270 19004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:30.276469 19004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:58:30.287007 19004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:30.288363 19004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:30.289705 19004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:30.303977 19009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:30.303858 19010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:30.303836 19012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:30.304574 19004 server_base.cc:1034] running on GCE node
I20260923 20:58:30.305271 19004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:30.306588 19004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:30.307824 19004 hybrid_clock.cc:648] HybridClock initialized: now 1790197110307769 us; error 53 us; skew 500 ppm
I20260923 20:58:30.308352 19004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:30 dist-test-slave-plcg krb5kdc[18630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:30.311952 19004 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:58:30.313508 19004 webserver.cc:460] Webserver started at http://127.0.97.3:42869/ using document root <none> and password file <none>
I20260923 20:58:30.314316 19004 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:30.314461 19004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:30.314798 19004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:30.318064 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "467bbb3f2f5042e2adf21698decf1e48"
format_stamp: "Formatted at 2026-09-23 20:58:30 on dist-test-slave-plcg"
server_key: "251df160d90c422e6e0d045aacb4d1f6"
server_key_iv: "e0076aab51dce0782a484cc3ed00618a"
server_key_version: "encryptionkey@0"
I20260923 20:58:30.318929 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "467bbb3f2f5042e2adf21698decf1e48"
format_stamp: "Formatted at 2026-09-23 20:58:30 on dist-test-slave-plcg"
server_key: "251df160d90c422e6e0d045aacb4d1f6"
server_key_iv: "e0076aab51dce0782a484cc3ed00618a"
server_key_version: "encryptionkey@0"
I20260923 20:58:30.324400 19004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:58:30.328094 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:30.329613 19004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:58:30.329885 19004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "467bbb3f2f5042e2adf21698decf1e48"
format_stamp: "Formatted at 2026-09-23 20:58:30 on dist-test-slave-plcg"
server_key: "251df160d90c422e6e0d045aacb4d1f6"
server_key_iv: "e0076aab51dce0782a484cc3ed00618a"
server_key_version: "encryptionkey@0"
I20260923 20:58:30.330123 19004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:30.343212 18864 heartbeater.cc:503] Master 127.0.97.62:41491 was elected leader, sending a full tablet report...
I20260923 20:58:30.354043 19004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:30.358469 19004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:30.358788 19004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:30.359817 19004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:30.361289 19004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:30.361406 19004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:30.361563 19004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:30.361639 19004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:30.396375 19004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44533
I20260923 20:58:30.396466 19131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44533 every 8 connection(s)
I20260923 20:58:30.397727 19004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:58:30.407636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19004
I20260923 20:58:30.407845 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:58:30.408363 388 external_mini_cluster.cc:1428] Setting key 0f37db4af326680444272e70869efbdc
Sep 23 20:58:30 dist-test-slave-plcg krb5kdc[18630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:30.418694 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.400415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59169 (local address 127.0.97.62:41491)
0923 20:58:30.400880 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.400884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.401756 (+ 872us) server_negotiation.cc:407] Connection header received
0923 20:58:30.403886 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.403891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.403992 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.404125 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:30.405838 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.406625 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.408595 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.409207 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.409368 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.413133 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:30.413139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:30.413142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:30.413189 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:30.415753 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.416531 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.416535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.416537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.416663 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.417048 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.417051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.417053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.417327 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:30.417439 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.418226 (+ 787us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.418418 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260923 20:58:30.419737 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.400839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41491 (local address 127.0.97.3:59169)
0923 20:58:30.401546 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:30.401627 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:30.403553 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:30.404383 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:30.404393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:30.405044 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:30.405622 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.405638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.406790 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.406796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.408419 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.408439 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.409470 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.409477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.409752 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.410518 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:30.410545 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:30.412909 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:30.415908 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:30.415916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:30.415932 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:30.416419 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:30.416760 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:30.416764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:30.416766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:30.416955 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:30.417436 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:30.417442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:30.417784 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 20:58:30.418108 (+ 324us) client_negotiation.cc:240] Negotiation successful
0923 20:58:30.418536 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":282,"threads_started":1}
I20260923 20:58:30.421212 19132 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41491
I20260923 20:58:30.421542 19132 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:30.422490 19132 heartbeater.cc:511] Master 127.0.97.62:41491 requested a full tablet report, sending...
I20260923 20:58:30.424021 18675 ts_manager.cc:194] Registered new tserver with Master: 467bbb3f2f5042e2adf21698decf1e48 (127.0.97.3:44533)
I20260923 20:58:30.424822 18675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59169
I20260923 20:58:30.436249 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:58:30.454118 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.440280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.0.97.62:41491)
0923 20:58:30.440443 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.440448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.440875 (+ 427us) server_negotiation.cc:407] Connection header received
0923 20:58:30.441033 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.441036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.441100 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.441203 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:30.442676 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.443426 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.444840 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.445268 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.445410 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.448290 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:30.448296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:30.448300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:30.448337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:30.451058 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.451928 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.451932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.451935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.452058 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.452540 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.452552 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.452555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.452874 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:30.453153 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.453548 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.453782 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:58:30.458501 18675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:58:30.465682 18675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:58:30.505065 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.479313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33919 (local address 127.0.0.1:54772)
0923 20:58:30.487757 (+ 8444us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:30.487819 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:30.488114 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:30.497119 (+ 9005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:30.497123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:30.497150 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:30.497645 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.497662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.499787 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.499794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.502172 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.502182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.503955 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.503963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.504197 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.504296 (+ 99us) client_negotiation.cc:769] Sending connection context
0923 20:58:30.504414 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 20:58:30.504646 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8052,"mutex_wait_us":67,"thread_start_us":15193,"threads_started":1}
I20260923 20:58:30.505164 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.483780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.0.97.3:44533)
0923 20:58:30.488963 (+ 5183us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.488989 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.496288 (+ 7299us) server_negotiation.cc:407] Connection header received
0923 20:58:30.496396 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.496403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.496661 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.496859 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:30.497918 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.499499 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.501641 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.503272 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.503411 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.503689 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.504597 (+ 908us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.504782 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4939,"thread_start_us":4749,"threads_started":1}
I20260923 20:58:30.505455 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.479655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43013 (local address 127.0.0.1:33136)
0923 20:58:30.494858 (+ 15203us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:30.494885 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:30.495057 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:30.496501 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:30.496508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:30.496543 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:30.496980 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.497000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.499077 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.499082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.501886 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.501902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.504139 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.504154 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.504317 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.504342 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:58:30.504414 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 20:58:30.504874 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15036,"mutex_wait_us":288,"thread_start_us":16036,"threads_started":1}
I20260923 20:58:30.505914 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.483002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44533 (local address 127.0.0.1:52548)
0923 20:58:30.495962 (+ 12960us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:30.495981 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:30.496105 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:30.497232 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:30.497237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:30.497267 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:30.497697 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.497714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.499659 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.499666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.501447 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.501461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.503845 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.503852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.504131 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.504277 (+ 146us) client_negotiation.cc:769] Sending connection context
0923 20:58:30.504414 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:58:30.505483 (+ 1069us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12825,"mutex_wait_us":885,"thread_start_us":4159,"threads_started":1}
I20260923 20:58:30.506193 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.480255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.0.97.2:43013)
0923 20:58:30.495559 (+ 15304us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.495572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.495708 (+ 136us) server_negotiation.cc:407] Connection header received
0923 20:58:30.495829 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.495838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.496143 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.496386 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:30.497179 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.498907 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.502068 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.503597 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.503788 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.503987 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.504579 (+ 592us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.505802 (+ 1223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15146,"thread_start_us":15029,"threads_started":1}
I20260923 20:58:30.506349 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.479646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.0.97.1:33919)
0923 20:58:30.496477 (+ 16831us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.496484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.496510 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:58:30.496609 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.496617 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.496879 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.497756 (+ 877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:30.498008 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.499535 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.502334 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.503403 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.503556 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.503795 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.505733 (+ 1938us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.505956 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16666,"thread_start_us":16492,"threads_started":1}
I20260923 20:58:30.509861 19067 tablet_service.cc:1467] Processing CreateTablet for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd (DEFAULT_TABLE table=test-table [id=e33c25c967ea44768a1f49718c615324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:30.509827 18799 tablet_service.cc:1467] Processing CreateTablet for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd (DEFAULT_TABLE table=test-table [id=e33c25c967ea44768a1f49718c615324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:30.511333 19067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:30.511425 18799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:30.512787 18933 tablet_service.cc:1467] Processing CreateTablet for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd (DEFAULT_TABLE table=test-table [id=e33c25c967ea44768a1f49718c615324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:30.514524 18933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:30.523301 19151 tablet_bootstrap.cc:492] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Bootstrap starting.
I20260923 20:58:30.526443 19152 tablet_bootstrap.cc:492] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Bootstrap starting.
I20260923 20:58:30.527190 19151 tablet_bootstrap.cc:654] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:30.528721 19151 log.cc:824] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:30.529712 19153 tablet_bootstrap.cc:492] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Bootstrap starting.
I20260923 20:58:30.530892 19152 tablet_bootstrap.cc:654] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:30.532202 19152 log.cc:824] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:30.533289 19151 tablet_bootstrap.cc:492] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: No bootstrap required, opened a new log
I20260923 20:58:30.533636 19151 ts_tablet_manager.cc:1397] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 20:58:30.534312 19153 tablet_bootstrap.cc:654] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:30.535459 19152 tablet_bootstrap.cc:492] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: No bootstrap required, opened a new log
I20260923 20:58:30.535655 19153 log.cc:824] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:30.535761 19152 ts_tablet_manager.cc:1397] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260923 20:58:30.539268 19153 tablet_bootstrap.cc:492] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: No bootstrap required, opened a new log
I20260923 20:58:30.539598 19153 ts_tablet_manager.cc:1397] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:58:30.544847 19152 raft_consensus.cc:348] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.544765 19151 raft_consensus.cc:348] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.545310 19152 raft_consensus.cc:374] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:30.545324 19151 raft_consensus.cc:374] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:30.545449 19151 raft_consensus.cc:729] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467bbb3f2f5042e2adf21698decf1e48, State: Initialized, Role: FOLLOWER
I20260923 20:58:30.545461 19152 raft_consensus.cc:729] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904c82fa81eb40ccb4dac0057dec7b4a, State: Initialized, Role: FOLLOWER
I20260923 20:58:30.546224 19152 consensus_queue.cc:260] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.546226 19151 consensus_queue.cc:260] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.548228 19152 ts_tablet_manager.cc:1428] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260923 20:58:30.549196 19132 heartbeater.cc:503] Master 127.0.97.62:41491 was elected leader, sending a full tablet report...
I20260923 20:58:30.550345 19151 ts_tablet_manager.cc:1428] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
I20260923 20:58:30.551540 19153 raft_consensus.cc:348] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.552035 19153 raft_consensus.cc:374] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:30.552184 19153 raft_consensus.cc:729] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c82eeddf4a64299947f973743824a79, State: Initialized, Role: FOLLOWER
I20260923 20:58:30.552924 19153 consensus_queue.cc:260] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.554579 18998 heartbeater.cc:503] Master 127.0.97.62:41491 was elected leader, sending a full tablet report...
I20260923 20:58:30.556946 19153 ts_tablet_manager.cc:1428] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260923 20:58:30.560747 18865 tablet.cc:2369] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:58:30.602010 18999 tablet.cc:2369] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:58:30.649441 19133 tablet.cc:2369] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:30.661552 19159 raft_consensus.cc:482] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:58:30.661926 19159 raft_consensus.cc:504] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.663661 19159 leader_election.cc:288] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 904c82fa81eb40ccb4dac0057dec7b4a (127.0.97.1:33919), 467bbb3f2f5042e2adf21698decf1e48 (127.0.97.3:44533)
I20260923 20:58:30.670818 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.664102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55187 (local address 127.0.97.3:44533)
0923 20:58:30.664400 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.664410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.664967 (+ 557us) server_negotiation.cc:407] Connection header received
0923 20:58:30.665209 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.665212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.665276 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.665383 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:30.666133 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.666881 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.668831 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.669641 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.669805 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.669963 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.670333 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.670470 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:58:30.670877 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.663922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44533 (local address 127.0.97.2:55187)
0923 20:58:30.664831 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:30.664849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:30.665050 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:30.665397 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:30.665402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:30.665427 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:30.665960 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.665971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.667025 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.667032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.668638 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.668654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.669942 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.669950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.670154 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.670180 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:58:30.670300 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 20:58:30.670447 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":32,"thread_start_us":816,"threads_started":1}
I20260923 20:58:30.671320 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.663995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52015 (local address 127.0.97.1:33919)
0923 20:58:30.664230 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.664239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.665178 (+ 939us) server_negotiation.cc:407] Connection header received
0923 20:58:30.665377 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.665381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.665441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.665542 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:30.666269 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.666978 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.668840 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.669661 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.669823 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.669964 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.670346 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.670553 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:58:30.671671 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.663993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33919 (local address 127.0.97.2:52015)
0923 20:58:30.665044 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:30.665058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:30.665216 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:30.665557 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:30.665561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:30.665597 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:30.666107 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.666117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.667126 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.667133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.668638 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:30.668661 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.669946 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.669953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:30.670178 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.670202 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:58:30.670312 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:58:30.671291 (+ 979us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":50,"thread_start_us":443,"threads_started":1}
I20260923 20:58:30.672092 19087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc0d2fa7b064ee8b2cb6cbdfca07cbd" candidate_uuid: "5c82eeddf4a64299947f973743824a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467bbb3f2f5042e2adf21698decf1e48" is_pre_election: true
I20260923 20:58:30.672729 19087 raft_consensus.cc:2454] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c82eeddf4a64299947f973743824a79 in term 0.
I20260923 20:58:30.672708 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc0d2fa7b064ee8b2cb6cbdfca07cbd" candidate_uuid: "5c82eeddf4a64299947f973743824a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" is_pre_election: true
I20260923 20:58:30.673266 18819 raft_consensus.cc:2454] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c82eeddf4a64299947f973743824a79 in term 0.
I20260923 20:58:30.673512 18888 leader_election.cc:302] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467bbb3f2f5042e2adf21698decf1e48, 5c82eeddf4a64299947f973743824a79; no voters:
I20260923 20:58:30.673960 19159 raft_consensus.cc:2781] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:58:30.674116 19159 raft_consensus.cc:482] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:58:30.674221 19159 raft_consensus.cc:3037] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:30.676435 19159 raft_consensus.cc:504] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.677078 19159 leader_election.cc:288] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [CANDIDATE]: Term 1 election: Requested vote from peers 904c82fa81eb40ccb4dac0057dec7b4a (127.0.97.1:33919), 467bbb3f2f5042e2adf21698decf1e48 (127.0.97.3:44533)
I20260923 20:58:30.677583 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc0d2fa7b064ee8b2cb6cbdfca07cbd" candidate_uuid: "5c82eeddf4a64299947f973743824a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904c82fa81eb40ccb4dac0057dec7b4a"
I20260923 20:58:30.677662 19087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc0d2fa7b064ee8b2cb6cbdfca07cbd" candidate_uuid: "5c82eeddf4a64299947f973743824a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "467bbb3f2f5042e2adf21698decf1e48"
I20260923 20:58:30.677861 18819 raft_consensus.cc:3037] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:30.677891 19087 raft_consensus.cc:3037] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:30.681186 18819 raft_consensus.cc:2454] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c82eeddf4a64299947f973743824a79 in term 1.
I20260923 20:58:30.681186 19087 raft_consensus.cc:2454] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c82eeddf4a64299947f973743824a79 in term 1.
I20260923 20:58:30.681701 18888 leader_election.cc:302] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467bbb3f2f5042e2adf21698decf1e48, 5c82eeddf4a64299947f973743824a79; no voters:
I20260923 20:58:30.682085 19159 raft_consensus.cc:2781] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:30.682531 19159 raft_consensus.cc:686] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 1 LEADER]: Becoming Leader. State: Replica: 5c82eeddf4a64299947f973743824a79, State: Running, Role: LEADER
I20260923 20:58:30.683089 19159 consensus_queue.cc:237] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } }
I20260923 20:58:30.688194 18675 catalog_manager.cc:5495] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c82eeddf4a64299947f973743824a79 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5c82eeddf4a64299947f973743824a79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c82eeddf4a64299947f973743824a79" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 } health_report { overall_health: UNKNOWN } } }
I20260923 20:58:30.743176 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.737334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.0.97.2:43013)
0923 20:58:30.737579 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.737587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.737611 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:58:30.737949 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.737956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.738100 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.738275 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:58:30.738764 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.739606 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.741035 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.741447 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.741554 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.742100 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:58:30.742638 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:58:30.742774 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.742913 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.742958 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:58:30.752156 19087 raft_consensus.cc:1261] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 1 FOLLOWER]: Refusing update from remote peer 5c82eeddf4a64299947f973743824a79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:30.752156 18819 raft_consensus.cc:1261] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 1 FOLLOWER]: Refusing update from remote peer 5c82eeddf4a64299947f973743824a79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:30.753222 19162 consensus_queue.cc:1035] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467bbb3f2f5042e2adf21698decf1e48" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:30.753579 19159 consensus_queue.cc:1035] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "904c82fa81eb40ccb4dac0057dec7b4a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:30.765532 19165 mvcc.cc:205] Tried to move back new op lower bound from 7332647365629657088 to 7332647365362708480. Current Snapshot: MvccSnapshot[applied={T|T < 7332647365629657088}]
I20260923 20:58:30.766253 19167 mvcc.cc:205] Tried to move back new op lower bound from 7332647365629657088 to 7332647365362708480. Current Snapshot: MvccSnapshot[applied={T|T < 7332647365629657088}]
I20260923 20:58:30.785941 18675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:30.786266 18675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:30.790539 18675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 228127d6672e4841b5eed25d1f19c6d0: Sending DeleteTablet for 3 replicas of tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd
I20260923 20:58:30.792029 18799 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54772
I20260923 20:58:30.792004 19067 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52548
I20260923 20:58:30.792894 18933 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33136
I20260923 20:58:30.796948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 18736
I20260923 20:58:30.815845 19177 tablet_replica.cc:330] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: stopping tablet replica
I20260923 20:58:30.816570 19177 raft_consensus.cc:2229] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:58:30.817503 19177 raft_consensus.cc:2258] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:30.818832 19175 tablet_replica.cc:330] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: stopping tablet replica
I20260923 20:58:30.819265 19175 raft_consensus.cc:2229] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:30.819828 19175 raft_consensus.cc:2258] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:30.820657 19176 tablet_replica.cc:330] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: stopping tablet replica
I20260923 20:58:30.821082 19176 raft_consensus.cc:2229] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:30.821482 19177 ts_tablet_manager.cc:1905] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:30.821615 19176 raft_consensus.cc:2258] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:30.824004 19175 ts_tablet_manager.cc:1905] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:30.823984 19176 ts_tablet_manager.cc:1905] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:58:30 dist-test-slave-plcg krb5kdc[18630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:58:30.829676 19177 ts_tablet_manager.cc:1918] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:30.829900 19177 log.cc:1196] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/bdc0d2fa7b064ee8b2cb6cbdfca07cbd
I20260923 20:58:30.830847 19177 ts_tablet_manager.cc:1939] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 5c82eeddf4a64299947f973743824a79: Deleting consensus metadata
I20260923 20:58:30.831593 19175 ts_tablet_manager.cc:1918] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:30.831772 19175 log.cc:1196] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/bdc0d2fa7b064ee8b2cb6cbdfca07cbd
I20260923 20:58:30.831875 19176 ts_tablet_manager.cc:1918] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:30.832049 19176 log.cc:1196] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790196790989944-388-0/minicluster-data/ts-0/wal/wals/bdc0d2fa7b064ee8b2cb6cbdfca07cbd
I20260923 20:58:30.832273 19175 ts_tablet_manager.cc:1939] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 467bbb3f2f5042e2adf21698decf1e48: Deleting consensus metadata
I20260923 20:58:30.832594 19176 ts_tablet_manager.cc:1939] T bdc0d2fa7b064ee8b2cb6cbdfca07cbd P 904c82fa81eb40ccb4dac0057dec7b4a: Deleting consensus metadata
I20260923 20:58:30.833205 18662 catalog_manager.cc:4841] TS 5c82eeddf4a64299947f973743824a79 (127.0.97.2:43013): tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd (table test-table [id=e33c25c967ea44768a1f49718c615324]) successfully deleted
I20260923 20:58:30.833927 18661 catalog_manager.cc:4841] TS 467bbb3f2f5042e2adf21698decf1e48 (127.0.97.3:44533): tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd (table test-table [id=e33c25c967ea44768a1f49718c615324]) successfully deleted
I20260923 20:58:30.835839 18659 catalog_manager.cc:4841] TS 904c82fa81eb40ccb4dac0057dec7b4a (127.0.97.1:33919): tablet bdc0d2fa7b064ee8b2cb6cbdfca07cbd (table test-table [id=e33c25c967ea44768a1f49718c615324]) successfully deleted
I20260923 20:58:30.837805 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.799008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.97.1:33919)
0923 20:58:30.799276 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.799286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.815233 (+ 15947us) server_negotiation.cc:407] Connection header received
0923 20:58:30.815759 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.815766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.815877 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.815967 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:30.818172 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.819383 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.822154 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.822875 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.822984 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.829198 (+ 6214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:30.829217 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:30.829238 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:30.829290 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:30.834528 (+ 5238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.835414 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.835424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.835428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.835607 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.836118 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.836123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.836127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.836612 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:30.836944 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.837161 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.837395 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 20:58:30.838570 18859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:30.950773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 18736
I20260923 20:58:30.969528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 18870
Sep 23 20:58:30 dist-test-slave-plcg krb5kdc[18630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:58:30.989286 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:30.970432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.0.97.2:43013)
0923 20:58:30.970633 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:30.970651 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:30.970680 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:58:30.970787 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:30.970793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:30.970888 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:30.971007 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:30.972285 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.973563 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.975363 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:30.976078 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:30.976249 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:30.981520 (+ 5271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:30.981543 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:30.981566 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:30.981623 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:30.986139 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.986974 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.986987 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.986992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.987184 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:30.987562 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:30.987568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:30.987572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:30.988073 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:30.988413 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:30.988608 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 20:58:30.988859 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:58:30.990276 18993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:31.108377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 18870
I20260923 20:58:31.127983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19004
Sep 23 20:58:31 dist-test-slave-plcg krb5kdc[18630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:58:31.143980 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:31.128746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.97.3:44533)
0923 20:58:31.128945 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:31.128954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:31.128996 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:58:31.129210 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:31.129216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:31.129298 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:31.129426 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:31.130910 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:31.131631 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:31.132973 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:31.133381 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:31.133526 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:31.138386 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:31.138404 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:31.138421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:31.138461 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:31.141340 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:31.142173 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:31.142181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:31.142184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:31.142342 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:31.142676 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:31.142679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:31.142681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:31.143026 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:31.143262 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:31.143529 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:58:31.143705 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:58:31.144724 19127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:31.235896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19004
I20260923 20:58:31.253078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 18643
I20260923 20:58:31.254642 18705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:31.368522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 18643
2026-09-23T20:58:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4582 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T20:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:31Z Disabled control of system clock
I20260923 20:58:31.447830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44317
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:58:31.795519 19186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:31.795838 19186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:31.795982 19186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:31.803661 19186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:31.803786 19186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:31.803877 19186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:31.803988 19186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:31.804050 19186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:31.813898 19186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:31.815300 19186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:31.817184 19186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:31.829847 19191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:31.829787 19194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:31.829794 19192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:31.830958 19186 server_base.cc:1034] running on GCE node
I20260923 20:58:31.831826 19186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:31.833293 19186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:31.834549 19186 hybrid_clock.cc:648] HybridClock initialized: now 1790197111834504 us; error 42 us; skew 500 ppm
I20260923 20:58:31.834980 19186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:31.837617 19186 webserver.cc:460] Webserver started at http://127.0.97.62:38973/ using document root <none> and password file <none>
I20260923 20:58:31.838425 19186 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:31.838546 19186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:31.838907 19186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:31.842147 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9"
format_stamp: "Formatted at 2026-09-23 20:58:31 on dist-test-slave-plcg"
server_key: "dce8c6c220027b5705ea41fa4969c32b"
server_key_iv: "53f493a41d8efecb134fdb139440431e"
server_key_version: "encryptionkey@0"
I20260923 20:58:31.843050 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9"
format_stamp: "Formatted at 2026-09-23 20:58:31 on dist-test-slave-plcg"
server_key: "dce8c6c220027b5705ea41fa4969c32b"
server_key_iv: "53f493a41d8efecb134fdb139440431e"
server_key_version: "encryptionkey@0"
I20260923 20:58:31.848703 19186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:58:31.852607 19200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:31.854220 19186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:58:31.854441 19186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9"
format_stamp: "Formatted at 2026-09-23 20:58:31 on dist-test-slave-plcg"
server_key: "dce8c6c220027b5705ea41fa4969c32b"
server_key_iv: "53f493a41d8efecb134fdb139440431e"
server_key_version: "encryptionkey@0"
I20260923 20:58:31.854696 19186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:31.876837 19186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:31.877763 19186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:31.878111 19186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:31.905762 19186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44317
I20260923 20:58:31.905781 19251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44317 every 8 connection(s)
I20260923 20:58:31.907301 19186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:31.911296 19252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:31.913789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19186
I20260923 20:58:31.914036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:31.914523 388 external_mini_cluster.cc:1428] Setting key f6c2ece80a28517d2fc06bd06343e901
I20260923 20:58:31.923735 19252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9: Bootstrap starting.
I20260923 20:58:31.927677 19252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:31.928853 19252 log.cc:824] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:31.929013 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:31.916577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.0.97.62:44317)
0923 20:58:31.917658 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:31.917677 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:31.917722 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:58:31.919510 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:31.919535 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:31.920058 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:31.920416 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:31.921052 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:31.922477 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:31.924396 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:31.925088 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:31.925222 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:31.926601 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:31.926608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:31.926624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:31.927174 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:31.927265 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:31.927286 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:31.927375 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:31.927637 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 20:58:31.927983 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":397,"threads_started":1}
I20260923 20:58:31.932504 19252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9: No bootstrap required, opened a new log
I20260923 20:58:31.941877 19252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44317 } }
I20260923 20:58:31.942330 19252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:31.942462 19252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfb6ce68f1843db99ce2a2234d4b6c9, State: Initialized, Role: FOLLOWER
I20260923 20:58:31.943138 19252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44317 } }
I20260923 20:58:31.943403 19252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:31.943571 19252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:31.943758 19252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:31.946707 19252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44317 } }
I20260923 20:58:31.947299 19252 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cfb6ce68f1843db99ce2a2234d4b6c9; no voters:
I20260923 20:58:31.947914 19252 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:31.948117 19257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:31.948881 19257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 9cfb6ce68f1843db99ce2a2234d4b6c9, State: Running, Role: LEADER
I20260923 20:58:31.949486 19257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44317 } }
I20260923 20:58:31.950173 19252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:31.953186 19258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44317 } } }
I20260923 20:58:31.953091 19259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfb6ce68f1843db99ce2a2234d4b6c9. Latest consensus state: current_term: 1 leader_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb6ce68f1843db99ce2a2234d4b6c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44317 } } }
I20260923 20:58:31.953647 19258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:31.953681 19259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:31.956527 19266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:31.962920 19266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:31.973591 19266 catalog_manager.cc:1347] Generated new cluster ID: a5580801ef4e47609e86baba5622fbea
I20260923 20:58:31.973780 19266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:31.999614 19266 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:32.001137 19266 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:32.013780 19266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9: Generated new TSK 0
I20260923 20:58:32.014796 19266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:32.028162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44317
--builtin_ntp_servers=127.0.97.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:58:32.389436 19276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:32.389818 19276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:32.390002 19276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:32.397574 19276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:32.397749 19276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:32.398023 19276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:58:32.407771 19276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:32.409147 19276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:32.411101 19276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:32.425427 19282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:32.425401 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:32.425468 19284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:32.426477 19276 server_base.cc:1034] running on GCE node
I20260923 20:58:32.427279 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:32.428434 19276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:32.429698 19276 hybrid_clock.cc:648] HybridClock initialized: now 1790197112429665 us; error 80 us; skew 500 ppm
I20260923 20:58:32.430114 19276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:32.432569 19276 webserver.cc:460] Webserver started at http://127.0.97.1:42933/ using document root <none> and password file <none>
I20260923 20:58:32.433336 19276 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:32.433461 19276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:32.433975 19276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:32.437372 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "063a68be74414bb4836431f780e8d1a4"
format_stamp: "Formatted at 2026-09-23 20:58:32 on dist-test-slave-plcg"
server_key: "d05f42720f71380707f94fed7f17451e"
server_key_iv: "6f90ae28bb92f4c5ea067e9be94a6550"
server_key_version: "encryptionkey@0"
I20260923 20:58:32.438300 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "063a68be74414bb4836431f780e8d1a4"
format_stamp: "Formatted at 2026-09-23 20:58:32 on dist-test-slave-plcg"
server_key: "d05f42720f71380707f94fed7f17451e"
server_key_iv: "6f90ae28bb92f4c5ea067e9be94a6550"
server_key_version: "encryptionkey@0"
I20260923 20:58:32.443799 19276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:58:32.447678 19290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:32.449193 19276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:58:32.449432 19276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "063a68be74414bb4836431f780e8d1a4"
format_stamp: "Formatted at 2026-09-23 20:58:32 on dist-test-slave-plcg"
server_key: "d05f42720f71380707f94fed7f17451e"
server_key_iv: "6f90ae28bb92f4c5ea067e9be94a6550"
server_key_version: "encryptionkey@0"
I20260923 20:58:32.449692 19276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:32.464594 19276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:32.465521 19276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:32.465874 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:32.466931 19276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:32.468744 19276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:32.468884 19276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:32.469039 19276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:32.469130 19276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:32.503520 19276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45931
I20260923 20:58:32.503578 19402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45931 every 8 connection(s)
I20260923 20:58:32.504972 19276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:58:32.508423 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19276
I20260923 20:58:32.508630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:58:32.509107 388 external_mini_cluster.cc:1428] Setting key fa756858255b122d2dd365c7553d6f34
I20260923 20:58:32.511914 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44317
--builtin_ntp_servers=127.0.97.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:58:32.520823 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:32.507800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50719 (local address 127.0.97.62:44317)
0923 20:58:32.508381 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:32.508395 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:32.509801 (+ 1406us) server_negotiation.cc:407] Connection header received
0923 20:58:32.511888 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:32.511896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:32.511989 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:32.512685 (+ 696us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:32.514123 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.514829 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.517477 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.517963 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.518079 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:32.519853 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:32.519858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:32.519861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:32.519882 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:32.519926 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:32.519939 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:32.520032 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:32.520369 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 20:58:32.520472 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":294,"threads_started":1}
I20260923 20:58:32.522037 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:32.508433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44317 (local address 127.0.97.1:50719)
0923 20:58:32.509598 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:32.509651 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:32.511592 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:32.512484 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:32.512500 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:32.512656 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:32.513903 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:32.513934 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.515007 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.515015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:32.517302 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:32.517323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.518209 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.518219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:32.518567 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:32.519408 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:32.519414 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:32.519476 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:32.519594 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:32.520063 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:32.520077 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:32.520088 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:58:32.520313 (+ 225us) client_negotiation.cc:240] Negotiation successful
0923 20:58:32.520813 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":422,"threads_started":1}
I20260923 20:58:32.523752 19403 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44317
I20260923 20:58:32.524127 19403 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:32.525125 19403 heartbeater.cc:511] Master 127.0.97.62:44317 requested a full tablet report, sending...
I20260923 20:58:32.527601 19217 ts_manager.cc:194] Registered new tserver with Master: 063a68be74414bb4836431f780e8d1a4 (127.0.97.1:45931)
I20260923 20:58:32.529150 19217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50719
W20260923 20:58:32.859625 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:32.859972 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:32.860137 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:32.867697 19407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:32.867867 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:32.868060 19407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:58:32.877630 19407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:32.879065 19407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:32.880878 19407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:32.894881 19412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:32.894909 19413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:32.894909 19415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:32.895813 19407 server_base.cc:1034] running on GCE node
I20260923 20:58:32.896553 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:32.897599 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:32.898856 19407 hybrid_clock.cc:648] HybridClock initialized: now 1790197112898826 us; error 52 us; skew 500 ppm
I20260923 20:58:32.899204 19407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:32.901661 19407 webserver.cc:460] Webserver started at http://127.0.97.2:41749/ using document root <none> and password file <none>
I20260923 20:58:32.902477 19407 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:32.902622 19407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:32.902971 19407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:32.906270 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "0d589dbd6ad1443c945d971e653c4f32"
format_stamp: "Formatted at 2026-09-23 20:58:32 on dist-test-slave-plcg"
server_key: "a4b532830035b08f87cf46b8f3845043"
server_key_iv: "0038afa2ab3ea6b02aac3706c4a856bd"
server_key_version: "encryptionkey@0"
I20260923 20:58:32.907132 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0d589dbd6ad1443c945d971e653c4f32"
format_stamp: "Formatted at 2026-09-23 20:58:32 on dist-test-slave-plcg"
server_key: "a4b532830035b08f87cf46b8f3845043"
server_key_iv: "0038afa2ab3ea6b02aac3706c4a856bd"
server_key_version: "encryptionkey@0"
I20260923 20:58:32.912935 19407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:58:32.916921 19421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:32.918530 19407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:32.918730 19407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "0d589dbd6ad1443c945d971e653c4f32"
format_stamp: "Formatted at 2026-09-23 20:58:32 on dist-test-slave-plcg"
server_key: "a4b532830035b08f87cf46b8f3845043"
server_key_iv: "0038afa2ab3ea6b02aac3706c4a856bd"
server_key_version: "encryptionkey@0"
I20260923 20:58:32.918994 19407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:32.943958 19407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:32.944865 19407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:32.945187 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:32.946233 19407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:32.947822 19407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:32.947968 19407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:32.948103 19407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:32.948190 19407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:32.983659 19407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42039
I20260923 20:58:32.983762 19533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42039 every 8 connection(s)
I20260923 20:58:32.985072 19407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:58:32.992924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19407
I20260923 20:58:32.993130 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:58:32.993564 388 external_mini_cluster.cc:1428] Setting key 8e9f18a92a1f9aa5ade56c92d9ae7a69
I20260923 20:58:32.996835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44317
--builtin_ntp_servers=127.0.97.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:58:32.999045 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:32.987859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37921 (local address 127.0.97.62:44317)
0923 20:58:32.988056 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:32.988060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:32.989506 (+ 1446us) server_negotiation.cc:407] Connection header received
0923 20:58:32.991561 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:32.991566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:32.991656 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:32.991752 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:32.992878 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.993793 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.995718 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.996335 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.996487 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:32.998073 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:32.998081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:32.998089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:32.998127 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:32.998173 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:32.998193 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:32.998295 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:32.998525 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 20:58:32.998648 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:58:32.999974 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:32.988349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44317 (local address 127.0.97.2:37921)
0923 20:58:32.989327 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:32.989366 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:32.991253 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:32.991958 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:32.991967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:32.992067 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:32.992661 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:32.992678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.993997 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.994001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:32.995540 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:32.995558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:32.996597 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:32.996603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:32.996825 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:32.997637 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:32.997640 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:32.997686 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:32.997791 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:32.998301 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:32.998310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:32.998316 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:58:32.998473 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 20:58:32.998821 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":319,"threads_started":1}
I20260923 20:58:33.001547 19534 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44317
I20260923 20:58:33.001931 19534 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:33.002902 19534 heartbeater.cc:511] Master 127.0.97.62:44317 requested a full tablet report, sending...
I20260923 20:58:33.004730 19217 ts_manager.cc:194] Registered new tserver with Master: 0d589dbd6ad1443c945d971e653c4f32 (127.0.97.2:42039)
I20260923 20:58:33.006076 19217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37921
W20260923 20:58:33.338253 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:33.338569 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:33.338662 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:33.346035 19537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:33.346150 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:33.346257 19537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:58:33.355865 19537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:33.357209 19537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:33.359122 19537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:33.373525 19542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:33.373507 19543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:33.374132 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:33.374696 19537 server_base.cc:1034] running on GCE node
I20260923 20:58:33.375360 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:33.376417 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:33.377633 19537 hybrid_clock.cc:648] HybridClock initialized: now 1790197113377584 us; error 42 us; skew 500 ppm
I20260923 20:58:33.378016 19537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:33.380532 19537 webserver.cc:460] Webserver started at http://127.0.97.3:35725/ using document root <none> and password file <none>
I20260923 20:58:33.381289 19537 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:33.381414 19537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:33.381764 19537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:33.384915 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "1049747b430949baa1f623ffe90c420b"
format_stamp: "Formatted at 2026-09-23 20:58:33 on dist-test-slave-plcg"
server_key: "6ef97f9c3ba27b48173275e8e6ed6817"
server_key_iv: "67f7a470bb7459efe028826db6f13506"
server_key_version: "encryptionkey@0"
I20260923 20:58:33.385784 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1049747b430949baa1f623ffe90c420b"
format_stamp: "Formatted at 2026-09-23 20:58:33 on dist-test-slave-plcg"
server_key: "6ef97f9c3ba27b48173275e8e6ed6817"
server_key_iv: "67f7a470bb7459efe028826db6f13506"
server_key_version: "encryptionkey@0"
I20260923 20:58:33.391877 19537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:58:33.395713 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:33.397083 19537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:58:33.397272 19537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "1049747b430949baa1f623ffe90c420b"
format_stamp: "Formatted at 2026-09-23 20:58:33 on dist-test-slave-plcg"
server_key: "6ef97f9c3ba27b48173275e8e6ed6817"
server_key_iv: "67f7a470bb7459efe028826db6f13506"
server_key_version: "encryptionkey@0"
I20260923 20:58:33.397511 19537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:33.424109 19537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:33.425031 19537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:33.425338 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:33.426433 19537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:33.428216 19537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:33.428357 19537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:33.428488 19537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:33.428586 19537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:33.463204 19537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45623
I20260923 20:58:33.463301 19663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45623 every 8 connection(s)
I20260923 20:58:33.464651 19537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:58:33.465113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19537
I20260923 20:58:33.465309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:58:33.465976 388 external_mini_cluster.cc:1428] Setting key 44d355b6118851623d185fc2ccc7423d
I20260923 20:58:33.478001 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.467609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43285 (local address 127.0.97.62:44317)
0923 20:58:33.467813 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.467817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.469449 (+ 1632us) server_negotiation.cc:407] Connection header received
0923 20:58:33.471456 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.471462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.471563 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.471674 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:33.472742 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.473474 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.475108 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.475526 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.475671 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.477101 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:33.477106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:33.477109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:33.477128 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:33.477161 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:33.477173 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:33.477264 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.477497 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.477614 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:58:33.478971 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.468278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44317 (local address 127.0.97.3:43285)
0923 20:58:33.469288 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:33.469336 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:33.471190 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:33.471907 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:33.471916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:33.472009 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:33.472578 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.472595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.473642 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.473647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.474943 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.474955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.475760 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.475767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.475999 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.476767 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:33.476769 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:33.476810 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:33.476891 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:33.477279 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:33.477291 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:33.477305 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 20:58:33.477482 (+ 177us) client_negotiation.cc:240] Negotiation successful
0923 20:58:33.477919 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":348,"threads_started":1}
I20260923 20:58:33.480423 19664 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44317
I20260923 20:58:33.480753 19664 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:33.481474 19664 heartbeater.cc:511] Master 127.0.97.62:44317 requested a full tablet report, sending...
I20260923 20:58:33.482887 19216 ts_manager.cc:194] Registered new tserver with Master: 1049747b430949baa1f623ffe90c420b (127.0.97.3:45623)
I20260923 20:58:33.483788 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43285
I20260923 20:58:33.493640 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:58:33.503845 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.497792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.0.97.62:44317)
0923 20:58:33.498068 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.498075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.498329 (+ 254us) server_negotiation.cc:407] Connection header received
0923 20:58:33.498462 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.498466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.498555 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.498636 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:33.499056 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.499768 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.501177 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.501732 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.501912 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.503032 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:33.503037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:33.503040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:33.503058 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:33.503094 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:33.503107 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:33.503216 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.503443 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.503531 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 20:58:33.508561 19216 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:58:33.516098 19216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:58:33.533449 19403 heartbeater.cc:503] Master 127.0.97.62:44317 was elected leader, sending a full tablet report...
I20260923 20:58:33.541802 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.532409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.0.97.2:42039)
0923 20:58:33.533203 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.533213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.533245 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:58:33.533345 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.533352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.533623 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.534041 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:33.534717 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.536352 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.539345 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.540486 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.540671 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.540960 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.541236 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.541420 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":472,"threads_started":1}
I20260923 20:58:33.541849 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.531893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42039 (local address 127.0.0.1:59754)
0923 20:58:33.532487 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:33.532506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:33.532627 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:33.533888 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:33.533892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:33.533920 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:33.534468 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.534488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.536520 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.536526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.539060 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.539106 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.540714 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.540721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.540943 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.541004 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 20:58:33.541188 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 20:58:33.541413 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":170,"thread_start_us":706,"threads_started":1}
I20260923 20:58:33.542753 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.529366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46924)
0923 20:58:33.530807 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:33.530849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:33.531073 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:33.532205 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:33.532217 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:33.532280 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:33.534721 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.534735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.536548 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.536565 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.539449 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.539469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.541908 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.541918 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.542109 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.542142 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:58:33.542223 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 20:58:33.542369 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":110,"thread_start_us":979,"threads_started":1}
I20260923 20:58:33.543622 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.529652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.0.97.1:45931)
0923 20:58:33.530298 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.530311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.530977 (+ 666us) server_negotiation.cc:407] Connection header received
0923 20:58:33.531311 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.531323 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.531598 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.531817 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:33.534974 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.536361 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.539697 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.541259 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.541426 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.541735 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.543058 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.543260 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":335,"threads_started":1}
I20260923 20:58:33.544581 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.531202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45623 (local address 127.0.0.1:58228)
0923 20:58:33.531821 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:33.531840 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:33.531951 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:33.536506 (+ 4555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:33.536514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:33.536545 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:33.537092 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.537111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.539174 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.539178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.541296 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.541313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.544047 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.544052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.544185 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.544210 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:58:33.544265 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 20:58:33.544362 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":314,"threads_started":1}
I20260923 20:58:33.545572 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.531534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.97.3:45623)
0923 20:58:33.535834 (+ 4300us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.535846 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.535881 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:58:33.535980 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.535988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.536268 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.536494 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:33.537324 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.538999 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.542273 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.543800 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.544864 (+ 1064us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.545112 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.545178 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.545304 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4126,"thread_start_us":3972,"threads_started":1}
I20260923 20:58:33.546792 19469 tablet_service.cc:1467] Processing CreateTablet for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c (DEFAULT_TABLE table=test-table [id=2adeb1a33d5f42f2a8822f1f1fa39765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:33.549111 19469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:33.548591 19338 tablet_service.cc:1467] Processing CreateTablet for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c (DEFAULT_TABLE table=test-table [id=2adeb1a33d5f42f2a8822f1f1fa39765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:33.549495 19599 tablet_service.cc:1467] Processing CreateTablet for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c (DEFAULT_TABLE table=test-table [id=2adeb1a33d5f42f2a8822f1f1fa39765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:33.550271 19338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:33.550897 19599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:33.564002 19682 tablet_bootstrap.cc:492] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Bootstrap starting.
I20260923 20:58:33.567725 19683 tablet_bootstrap.cc:492] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: Bootstrap starting.
I20260923 20:58:33.567729 19684 tablet_bootstrap.cc:492] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Bootstrap starting.
I20260923 20:58:33.568255 19682 tablet_bootstrap.cc:654] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:33.569706 19682 log.cc:824] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:33.571663 19684 tablet_bootstrap.cc:654] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:33.571714 19683 tablet_bootstrap.cc:654] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:33.572885 19683 log.cc:824] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:33.572978 19684 log.cc:824] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:33.574196 19682 tablet_bootstrap.cc:492] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: No bootstrap required, opened a new log
I20260923 20:58:33.574537 19682 ts_tablet_manager.cc:1397] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 20:58:33.575845 19683 tablet_bootstrap.cc:492] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: No bootstrap required, opened a new log
I20260923 20:58:33.576184 19683 ts_tablet_manager.cc:1397] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:58:33.576244 19684 tablet_bootstrap.cc:492] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: No bootstrap required, opened a new log
I20260923 20:58:33.576543 19684 ts_tablet_manager.cc:1397] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260923 20:58:33.584149 19683 raft_consensus.cc:348] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.584613 19683 raft_consensus.cc:374] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:33.584738 19683 raft_consensus.cc:729] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063a68be74414bb4836431f780e8d1a4, State: Initialized, Role: FOLLOWER
I20260923 20:58:33.585441 19683 consensus_queue.cc:260] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.585911 19682 raft_consensus.cc:348] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.586438 19682 raft_consensus.cc:374] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:33.586575 19682 raft_consensus.cc:729] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1049747b430949baa1f623ffe90c420b, State: Initialized, Role: FOLLOWER
I20260923 20:58:33.587405 19683 ts_tablet_manager.cc:1428] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260923 20:58:33.587380 19682 consensus_queue.cc:260] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.587539 19684 raft_consensus.cc:348] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.587985 19684 raft_consensus.cc:374] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:33.588188 19684 raft_consensus.cc:729] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d589dbd6ad1443c945d971e653c4f32, State: Initialized, Role: FOLLOWER
I20260923 20:58:33.588762 19664 heartbeater.cc:503] Master 127.0.97.62:44317 was elected leader, sending a full tablet report...
I20260923 20:58:33.588984 19684 consensus_queue.cc:260] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.589569 19682 ts_tablet_manager.cc:1428] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 20:58:33.591599 19534 heartbeater.cc:503] Master 127.0.97.62:44317 was elected leader, sending a full tablet report...
I20260923 20:58:33.593900 19684 ts_tablet_manager.cc:1428] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260923 20:58:33.683095 19688 raft_consensus.cc:482] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:58:33.683491 19688 raft_consensus.cc:504] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.685248 19688 leader_election.cc:288] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1049747b430949baa1f623ffe90c420b (127.0.97.3:45623), 0d589dbd6ad1443c945d971e653c4f32 (127.0.97.2:42039)
I20260923 20:58:33.694315 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.686814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47877 (local address 127.0.97.2:42039)
0923 20:58:33.686998 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.687004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.687626 (+ 622us) server_negotiation.cc:407] Connection header received
0923 20:58:33.687840 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.687846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.687936 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.688035 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:33.688513 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.689388 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.691828 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.693033 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.693151 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.693262 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.693905 (+ 643us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.694047 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:58:33.694315 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.685743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43357 (local address 127.0.97.3:45623)
0923 20:58:33.686009 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.686015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.686561 (+ 546us) server_negotiation.cc:407] Connection header received
0923 20:58:33.686903 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.686912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.687044 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.687122 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:33.687932 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.688929 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.691690 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.692997 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.693152 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.693283 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.693907 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.694052 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:58:33.694557 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.685717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45623 (local address 127.0.97.1:43357)
0923 20:58:33.686456 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:33.686478 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:33.686689 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:33.687335 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:33.687339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:33.687370 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:33.687806 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.687818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.689087 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.689090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.691439 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.691456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.693295 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.693301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.693629 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.693743 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 20:58:33.693870 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 20:58:33.694110 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":222,"thread_start_us":411,"threads_started":1}
I20260923 20:58:33.694576 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.686606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42039 (local address 127.0.97.1:47877)
0923 20:58:33.687507 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:33.687525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:33.687680 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:33.688062 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:33.688065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:33.688086 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:33.688350 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.688357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.689555 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.689560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.691635 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:33.691648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.693261 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.693274 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:33.693629 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.693743 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 20:58:33.693870 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 20:58:33.694110 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":203,"thread_start_us":560,"threads_started":1}
I20260923 20:58:33.695777 19619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dca73c9ec754e5a8ae8a5419e4d4d5c" candidate_uuid: "063a68be74414bb4836431f780e8d1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049747b430949baa1f623ffe90c420b" is_pre_election: true
I20260923 20:58:33.695801 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dca73c9ec754e5a8ae8a5419e4d4d5c" candidate_uuid: "063a68be74414bb4836431f780e8d1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d589dbd6ad1443c945d971e653c4f32" is_pre_election: true
I20260923 20:58:33.696372 19619 raft_consensus.cc:2454] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063a68be74414bb4836431f780e8d1a4 in term 0.
I20260923 20:58:33.696393 19489 raft_consensus.cc:2454] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063a68be74414bb4836431f780e8d1a4 in term 0.
I20260923 20:58:33.697160 19294 leader_election.cc:302] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063a68be74414bb4836431f780e8d1a4, 1049747b430949baa1f623ffe90c420b; no voters:
I20260923 20:58:33.697570 19688 raft_consensus.cc:2781] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:58:33.697724 19688 raft_consensus.cc:482] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:58:33.697860 19688 raft_consensus.cc:3037] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:33.700136 19688 raft_consensus.cc:504] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.700798 19688 leader_election.cc:288] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [CANDIDATE]: Term 1 election: Requested vote from peers 1049747b430949baa1f623ffe90c420b (127.0.97.3:45623), 0d589dbd6ad1443c945d971e653c4f32 (127.0.97.2:42039)
I20260923 20:58:33.701449 19619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dca73c9ec754e5a8ae8a5419e4d4d5c" candidate_uuid: "063a68be74414bb4836431f780e8d1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1049747b430949baa1f623ffe90c420b"
I20260923 20:58:33.701592 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dca73c9ec754e5a8ae8a5419e4d4d5c" candidate_uuid: "063a68be74414bb4836431f780e8d1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d589dbd6ad1443c945d971e653c4f32"
I20260923 20:58:33.701694 19619 raft_consensus.cc:3037] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:33.701900 19489 raft_consensus.cc:3037] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:33.704306 19619 raft_consensus.cc:2454] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063a68be74414bb4836431f780e8d1a4 in term 1.
I20260923 20:58:33.704324 19489 raft_consensus.cc:2454] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063a68be74414bb4836431f780e8d1a4 in term 1.
I20260923 20:58:33.704869 19294 leader_election.cc:302] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063a68be74414bb4836431f780e8d1a4, 1049747b430949baa1f623ffe90c420b; no voters:
I20260923 20:58:33.705201 19688 raft_consensus.cc:2781] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:33.705662 19688 raft_consensus.cc:686] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 1 LEADER]: Becoming Leader. State: Replica: 063a68be74414bb4836431f780e8d1a4, State: Running, Role: LEADER
I20260923 20:58:33.706276 19688 consensus_queue.cc:237] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } }
I20260923 20:58:33.711390 19215 catalog_manager.cc:5495] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 063a68be74414bb4836431f780e8d1a4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "063a68be74414bb4836431f780e8d1a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063a68be74414bb4836431f780e8d1a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 } health_report { overall_health: UNKNOWN } } }
W20260923 20:58:33.716534 19665 tablet.cc:2369] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:58:33.737258 19535 tablet.cc:2369] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:33.738186 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.732050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.0.97.1:45931)
0923 20:58:33.732274 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.732280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.732385 (+ 105us) server_negotiation.cc:407] Connection header received
0923 20:58:33.732654 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.732659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.732750 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.732914 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:58:33.733309 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.734089 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.735520 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.735948 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.736086 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.736726 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:58:33.737433 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:58:33.737631 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.737853 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.737902 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:58:33.747335 19489 raft_consensus.cc:1261] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 1 FOLLOWER]: Refusing update from remote peer 063a68be74414bb4836431f780e8d1a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:33.747336 19619 raft_consensus.cc:1261] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 1 FOLLOWER]: Refusing update from remote peer 063a68be74414bb4836431f780e8d1a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:33.748512 19688 consensus_queue.cc:1035] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1049747b430949baa1f623ffe90c420b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:33.748926 19693 consensus_queue.cc:1035] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d589dbd6ad1443c945d971e653c4f32" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:58:33.757447 19404 tablet.cc:2369] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:33.760193 19696 mvcc.cc:205] Tried to move back new op lower bound from 7332647377897267200 to 7332647377745575936. Current Snapshot: MvccSnapshot[applied={T|T < 7332647377897267200}]
I20260923 20:58:33.779409 19216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:33.779747 19216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:33.784437 19216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cfb6ce68f1843db99ce2a2234d4b6c9: Sending DeleteTablet for 3 replicas of tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c
I20260923 20:58:33.786017 19599 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:33 UTC) from {username='slave'} at 127.0.0.1:58228
I20260923 20:58:33.786017 19469 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:33 UTC) from {username='slave'} at 127.0.0.1:59754
I20260923 20:58:33.786013 19338 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:33 UTC) from {username='slave'} at 127.0.0.1:46924
I20260923 20:58:33.787303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19276
I20260923 20:58:33.798592 19706 tablet_replica.cc:330] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: stopping tablet replica
I20260923 20:58:33.799455 19706 raft_consensus.cc:2229] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:33.800314 19706 raft_consensus.cc:2258] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:33.801422 19707 tablet_replica.cc:330] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: stopping tablet replica
I20260923 20:58:33.804061 19706 ts_tablet_manager.cc:1905] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:33.804798 19707 raft_consensus.cc:2229] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:33.805567 19707 raft_consensus.cc:2258] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:33.808538 19708 tablet_replica.cc:330] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: stopping tablet replica
I20260923 20:58:33.809674 19708 raft_consensus.cc:2229] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:58:33.810485 19707 ts_tablet_manager.cc:1905] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:33.810485 19708 raft_consensus.cc:2258] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:33.813771 19706 ts_tablet_manager.cc:1918] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:33.814042 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.789066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.0.97.1:45931)
0923 20:58:33.789307 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.789317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.800654 (+ 11337us) server_negotiation.cc:407] Connection header received
0923 20:58:33.801490 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.801497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.801637 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.801727 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:33.802582 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.803841 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.805880 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.806618 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.806733 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.812135 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:33.812148 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:33.812166 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:33.812975 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:33.813053 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:33.813082 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:33.813173 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.813451 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.813577 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:58:33.814046 19706 log.cc:1196] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/2dca73c9ec754e5a8ae8a5419e4d4d5c
I20260923 20:58:33.814585 19706 ts_tablet_manager.cc:1939] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 1049747b430949baa1f623ffe90c420b: Deleting consensus metadata
I20260923 20:58:33.815629 19398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:33.816787 19204 catalog_manager.cc:4841] TS 1049747b430949baa1f623ffe90c420b (127.0.97.3:45623): tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c (table test-table [id=2adeb1a33d5f42f2a8822f1f1fa39765]) successfully deleted
I20260923 20:58:33.817751 19707 ts_tablet_manager.cc:1918] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:33.817996 19707 log.cc:1196] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/2dca73c9ec754e5a8ae8a5419e4d4d5c
I20260923 20:58:33.818610 19707 ts_tablet_manager.cc:1939] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 0d589dbd6ad1443c945d971e653c4f32: Deleting consensus metadata
I20260923 20:58:33.820549 19204 catalog_manager.cc:4841] TS 0d589dbd6ad1443c945d971e653c4f32 (127.0.97.2:42039): tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c (table test-table [id=2adeb1a33d5f42f2a8822f1f1fa39765]) successfully deleted
I20260923 20:58:33.817952 19708 ts_tablet_manager.cc:1905] T 2dca73c9ec754e5a8ae8a5419e4d4d5c P 063a68be74414bb4836431f780e8d1a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:33.938778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19276
I20260923 20:58:33.959262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19407
W20260923 20:58:33.959336 19204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45931 (error 108)
I20260923 20:58:33.961592 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.961086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46940)
0923 20:58:33.961264 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:33.961403 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 20:58:33.961876 19204 catalog_manager.cc:4584] TS 063a68be74414bb4836431f780e8d1a4 (127.0.97.1:45931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2dca73c9ec754e5a8ae8a5419e4d4d5c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
I20260923 20:58:33.966739 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:33.960123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.0.97.2:42039)
0923 20:58:33.960333 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:33.960339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:33.960492 (+ 153us) server_negotiation.cc:407] Connection header received
0923 20:58:33.960657 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:33.960662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:33.960788 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:33.960921 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:33.961411 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.962393 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.963650 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:33.964045 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:33.964155 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:33.965091 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:33.965106 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:33.965121 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:33.965706 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:33.965755 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:33.965775 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:33.966133 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:33.966267 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 20:58:33.966364 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:58:33.967549 19529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:34.015503 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:34.014993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46942)
0923 20:58:34.015226 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:34.015334 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 20:58:34.016607 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:34.016183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46954)
0923 20:58:34.016340 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:34.016420 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 20:58:34.057664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19407
I20260923 20:58:34.076319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19537
I20260923 20:58:34.078084 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:34.077503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46968)
0923 20:58:34.077712 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:34.077870 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 20:58:34.079514 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:34.079079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46974)
0923 20:58:34.079261 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:34.079372 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 20:58:34.084915 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:34.077122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.0.97.3:45623)
0923 20:58:34.077373 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:34.077382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:34.077409 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:58:34.077602 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:34.077607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:34.077755 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:34.078087 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:34.078568 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:34.079581 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:34.080986 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:34.081788 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:34.081980 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:34.083094 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:34.083105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:34.083129 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:34.084005 (+ 876us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:34.084082 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:34.084114 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:34.084252 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:34.084337 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 20:58:34.084477 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:58:34.085655 19659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:34.160355 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:34.159654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46976)
0923 20:58:34.160022 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:34.160136 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260923 20:58:34.161679 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:34.161197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45931 (local address 127.0.0.1:46980)
0923 20:58:34.161439 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:34.161526 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260923 20:58:34.180087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19537
I20260923 20:58:34.197162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19186
I20260923 20:58:34.198367 19247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:34.311734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19186
2026-09-23T20:58:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:34 dist-test-slave-plcg krb5kdc[19717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:34 dist-test-slave-plcg krb5kdc[19717](info): set up 2 sockets
Sep 23 20:58:34 dist-test-slave-plcg krb5kdc[19717](info): commencing operation
krb5kdc: starting...
W20260923 20:58:35.434054 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:35 dist-test-slave-plcg krb5kdc[19717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:58:35.698261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35673
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:36.045995 19730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:36.046321 19730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:36.046413 19730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:36.053889 19730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:36.054023 19730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:36.054073 19730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:36.054138 19730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:36.054185 19730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:36.064446 19730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:36.065794 19730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:36.067147 19730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:36.079694 19735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:36.079746 19738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:36.079659 19736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:36.081202 19730 server_base.cc:1034] running on GCE node
I20260923 20:58:36.082261 19730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:36.083894 19730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:36.085125 19730 hybrid_clock.cc:648] HybridClock initialized: now 1790197116085076 us; error 38 us; skew 500 ppm
I20260923 20:58:36.085531 19730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:36 dist-test-slave-plcg krb5kdc[19717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:36.089164 19730 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:58:36.091135 19730 webserver.cc:460] Webserver started at http://127.0.97.62:44665/ using document root <none> and password file <none>
I20260923 20:58:36.091969 19730 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:36.092124 19730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:36.092521 19730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:36.096051 19730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "8548456554a4415fbf9ea13a16062e4e"
format_stamp: "Formatted at 2026-09-23 20:58:36 on dist-test-slave-plcg"
server_key: "82e5e5949520cd36a5b8d7b6a5101a29"
server_key_iv: "576113ed2a61317358ff4d0d366870fe"
server_key_version: "encryptionkey@0"
I20260923 20:58:36.097036 19730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "8548456554a4415fbf9ea13a16062e4e"
format_stamp: "Formatted at 2026-09-23 20:58:36 on dist-test-slave-plcg"
server_key: "82e5e5949520cd36a5b8d7b6a5101a29"
server_key_iv: "576113ed2a61317358ff4d0d366870fe"
server_key_version: "encryptionkey@0"
I20260923 20:58:36.103333 19730 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:58:36.107784 19745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:36.109439 19730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:58:36.109675 19730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "8548456554a4415fbf9ea13a16062e4e"
format_stamp: "Formatted at 2026-09-23 20:58:36 on dist-test-slave-plcg"
server_key: "82e5e5949520cd36a5b8d7b6a5101a29"
server_key_iv: "576113ed2a61317358ff4d0d366870fe"
server_key_version: "encryptionkey@0"
I20260923 20:58:36.109983 19730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:36.131829 19730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:36.136659 19730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:36.137025 19730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:36.157119 19730 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35673
I20260923 20:58:36.157159 19796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35673 every 8 connection(s)
I20260923 20:58:36.158675 19730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:36.162698 19797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:36.164649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19730
I20260923 20:58:36.164844 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:36.165405 388 external_mini_cluster.cc:1428] Setting key a8cfcfbebf0ae71c8f92fd9c8f3a3003
I20260923 20:58:36.176606 19797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e: Bootstrap starting.
I20260923 20:58:36.181417 19797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e: Neither blocks nor log segments found. Creating new log.
Sep 23 20:58:36 dist-test-slave-plcg krb5kdc[19717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:36.183028 19797 log.cc:824] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:36.187098 19797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e: No bootstrap required, opened a new log
I20260923 20:58:36.192574 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:36.167529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.0.97.62:35673)
0923 20:58:36.168440 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:36.168455 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:36.168502 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:58:36.170419 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:36.170444 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:36.170878 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:36.171259 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:36.173281 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:36.174847 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:36.177014 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:36.177772 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:36.178005 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:36.183784 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:36.183797 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:36.183825 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:36.183907 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:36.187622 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:36.188707 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:36.188714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:36.188720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:36.188908 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:36.189461 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:36.189465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:36.189467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:36.190176 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:36.190450 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:36.190965 (+ 515us) server_negotiation.cc:299] Negotiation successful
0923 20:58:36.191406 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":421,"threads_started":1}
I20260923 20:58:36.197314 19797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8548456554a4415fbf9ea13a16062e4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260923 20:58:36.197765 19797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:36.197930 19797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8548456554a4415fbf9ea13a16062e4e, State: Initialized, Role: FOLLOWER
I20260923 20:58:36.198529 19797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8548456554a4415fbf9ea13a16062e4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260923 20:58:36.198793 19797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:36.198956 19797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:36.199136 19797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:36.201556 19797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8548456554a4415fbf9ea13a16062e4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260923 20:58:36.202199 19797 leader_election.cc:302] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8548456554a4415fbf9ea13a16062e4e; no voters:
I20260923 20:58:36.202817 19797 leader_election.cc:288] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:36.202984 19802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:36.203769 19802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [term 1 LEADER]: Becoming Leader. State: Replica: 8548456554a4415fbf9ea13a16062e4e, State: Running, Role: LEADER
I20260923 20:58:36.204375 19802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8548456554a4415fbf9ea13a16062e4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } }
I20260923 20:58:36.205103 19797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:36.207338 19804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8548456554a4415fbf9ea13a16062e4e. Latest consensus state: current_term: 1 leader_uuid: "8548456554a4415fbf9ea13a16062e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8548456554a4415fbf9ea13a16062e4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } } }
I20260923 20:58:36.207916 19804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:36.207365 19803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8548456554a4415fbf9ea13a16062e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8548456554a4415fbf9ea13a16062e4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35673 } } }
I20260923 20:58:36.208576 19803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:36.210041 19809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:36.215325 19809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:36.226199 19809 catalog_manager.cc:1347] Generated new cluster ID: d8f6e884c9bb496398e751d2f5fea65a
I20260923 20:58:36.226374 19809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:36.255514 19809 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:36.256990 19809 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:36.274153 19809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e: Generated new TSK 0
I20260923 20:58:36.275429 19809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 20:58:36.338901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:36.690905 19823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:36.691224 19823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:36.691442 19823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:36.699414 19823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:36.699607 19823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:36.699793 19823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:58:36.710491 19823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:36.711905 19823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:36.713480 19823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:36.728116 19828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:36.728183 19831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:36.728226 19829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:36.729430 19823 server_base.cc:1034] running on GCE node
I20260923 20:58:36.730585 19823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:36.732052 19823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:36.733304 19823 hybrid_clock.cc:648] HybridClock initialized: now 1790197116733262 us; error 43 us; skew 500 ppm
I20260923 20:58:36.733616 19823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:36 dist-test-slave-plcg krb5kdc[19717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:36.737061 19823 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 20:58:36.738849 19823 webserver.cc:460] Webserver started at http://127.0.97.1:44881/ using document root <none> and password file <none>
I20260923 20:58:36.739686 19823 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:36.739863 19823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:36.740228 19823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:36.743697 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "c38e7207da3d4ebaa3772e04205748f7"
format_stamp: "Formatted at 2026-09-23 20:58:36 on dist-test-slave-plcg"
server_key: "8d13bae1a4a0c87af8094cd01c080017"
server_key_iv: "0f7f033c803288e5226e0fe18ed864c3"
server_key_version: "encryptionkey@0"
I20260923 20:58:36.744684 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c38e7207da3d4ebaa3772e04205748f7"
format_stamp: "Formatted at 2026-09-23 20:58:36 on dist-test-slave-plcg"
server_key: "8d13bae1a4a0c87af8094cd01c080017"
server_key_iv: "0f7f033c803288e5226e0fe18ed864c3"
server_key_version: "encryptionkey@0"
I20260923 20:58:36.750972 19823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 20:58:36.755087 19838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:36.756635 19823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260923 20:58:36.756855 19823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "c38e7207da3d4ebaa3772e04205748f7"
format_stamp: "Formatted at 2026-09-23 20:58:36 on dist-test-slave-plcg"
server_key: "8d13bae1a4a0c87af8094cd01c080017"
server_key_iv: "0f7f033c803288e5226e0fe18ed864c3"
server_key_version: "encryptionkey@0"
I20260923 20:58:36.757094 19823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:36.787309 19823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:36.791972 19823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:36.792305 19823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:36.793375 19823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:36.794932 19823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:36.795051 19823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:36.795209 19823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:36.795286 19823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:36.831817 19823 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42477
I20260923 20:58:36.831918 19950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42477 every 8 connection(s)
I20260923 20:58:36.833226 19823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:58:36.835449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19823
I20260923 20:58:36.835661 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:58:36.836266 388 external_mini_cluster.cc:1428] Setting key a73990cb8e8ae250d22366fa36222a3d
Sep 23 20:58:36 dist-test-slave-plcg krb5kdc[19717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:36.857760 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:36.836187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35301 (local address 127.0.97.62:35673)
0923 20:58:36.836868 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:36.836875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:36.838058 (+ 1183us) server_negotiation.cc:407] Connection header received
0923 20:58:36.840136 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:36.840145 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:36.840244 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:36.840383 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:36.842162 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:36.843348 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:36.845195 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:36.846000 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:36.846165 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:36.850973 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:36.850979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:36.850983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:36.851026 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:36.854003 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:36.855131 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:36.855135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:36.855137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:36.855288 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:36.855837 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:36.855841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:36.855843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:36.856166 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:36.856292 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:36.857275 (+ 983us) server_negotiation.cc:299] Negotiation successful
0923 20:58:36.857483 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":380,"threads_started":1}
I20260923 20:58:36.859391 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:36.836897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35673 (local address 127.0.97.1:35301)
0923 20:58:36.837873 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:36.837932 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:36.839803 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:36.840612 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:36.840623 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:36.841238 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:36.841955 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:36.841979 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:36.843580 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:36.843584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:36.845004 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:36.845020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:36.846236 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:36.846241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:36.846461 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:36.847282 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:36.847318 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:36.850645 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:36.854196 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:36.854210 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:36.854233 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:36.854974 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:36.855423 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:36.855429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:36.855435 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:36.855704 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:36.856309 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:36.856321 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:36.856829 (+ 508us) client_negotiation.cc:769] Sending connection context
0923 20:58:36.857157 (+ 328us) client_negotiation.cc:240] Negotiation successful
0923 20:58:36.857779 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":364,"threads_started":1}
I20260923 20:58:36.861310 19951 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35673
I20260923 20:58:36.861773 19951 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:36.862664 19951 heartbeater.cc:511] Master 127.0.97.62:35673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 20:58:36.865016 19762 ts_manager.cc:194] Registered new tserver with Master: c38e7207da3d4ebaa3772e04205748f7 (127.0.97.1:42477)
I20260923 20:58:36.866616 19762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 20:58:36.897550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:37.238489 19957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:37.238806 19957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:37.238986 19957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:37.246650 19957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:37.246788 19957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:37.246980 19957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:58:37.257210 19957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:37.258615 19957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:37.260497 19957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:37.275381 19965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:37.275553 19962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:37.276394 19963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:37.281726 19957 server_base.cc:1034] running on GCE node
I20260923 20:58:37.282514 19957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:37.283617 19957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:37.284897 19957 hybrid_clock.cc:648] HybridClock initialized: now 1790197117284819 us; error 63 us; skew 500 ppm
I20260923 20:58:37.285374 19957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:37 dist-test-slave-plcg krb5kdc[19717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:37.288992 19957 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 20:58:37.290887 19957 webserver.cc:460] Webserver started at http://127.0.97.2:46321/ using document root <none> and password file <none>
I20260923 20:58:37.291697 19957 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:37.291852 19957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:37.292186 19957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:37.295353 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "b1003d805eb74a5aa45ff545cdc38604"
format_stamp: "Formatted at 2026-09-23 20:58:37 on dist-test-slave-plcg"
server_key: "d69679b1c98e96442fc95fb387d282b9"
server_key_iv: "d65622a24fcfc363ac447fb079227b6b"
server_key_version: "encryptionkey@0"
I20260923 20:58:37.296222 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b1003d805eb74a5aa45ff545cdc38604"
format_stamp: "Formatted at 2026-09-23 20:58:37 on dist-test-slave-plcg"
server_key: "d69679b1c98e96442fc95fb387d282b9"
server_key_iv: "d65622a24fcfc363ac447fb079227b6b"
server_key_version: "encryptionkey@0"
I20260923 20:58:37.301689 19957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:58:37.305531 19972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:37.307117 19957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:58:37.307322 19957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "b1003d805eb74a5aa45ff545cdc38604"
format_stamp: "Formatted at 2026-09-23 20:58:37 on dist-test-slave-plcg"
server_key: "d69679b1c98e96442fc95fb387d282b9"
server_key_iv: "d65622a24fcfc363ac447fb079227b6b"
server_key_version: "encryptionkey@0"
I20260923 20:58:37.307557 19957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:37.322892 19957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:37.327642 19957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:37.327970 19957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:37.329011 19957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:37.330577 19957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:37.330682 19957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:37.330823 19957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:37.330899 19957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:37.365983 19957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35171
I20260923 20:58:37.366029 20084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35171 every 8 connection(s)
I20260923 20:58:37.367353 19957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:58:37.374150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 19957
I20260923 20:58:37.374343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:58:37.374797 388 external_mini_cluster.cc:1428] Setting key fcbc539be3a4bc6e05e37599adf8a893
Sep 23 20:58:37 dist-test-slave-plcg krb5kdc[19717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:37.389026 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:37.369978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46949 (local address 127.0.97.62:35673)
0923 20:58:37.370497 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:37.370502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:37.371530 (+ 1028us) server_negotiation.cc:407] Connection header received
0923 20:58:37.373450 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:37.373456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:37.373552 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:37.373678 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:37.375604 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.376502 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.378312 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.378773 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.378921 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:37.382681 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:37.382686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:37.382689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:37.382724 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:37.385740 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:37.386596 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:37.386600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:37.386603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:37.386739 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:37.387212 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:37.387216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:37.387219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:37.387503 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:37.387665 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:37.388391 (+ 726us) server_negotiation.cc:299] Negotiation successful
0923 20:58:37.388634 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260923 20:58:37.390134 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:37.370572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35673 (local address 127.0.97.2:46949)
0923 20:58:37.371346 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:37.371391 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:37.373146 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:37.373991 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:37.374000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:37.374659 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:37.375359 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:37.375386 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.376679 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.376686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:37.378149 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:37.378162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.379020 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.379028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:37.379366 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:37.380091 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:37.380114 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:37.382404 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:37.385945 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:37.385955 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:37.385982 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:37.386443 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:37.386865 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:37.386879 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:37.386882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:37.387079 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:37.387634 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:37.387642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:37.388014 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 20:58:37.388283 (+ 269us) client_negotiation.cc:240] Negotiation successful
0923 20:58:37.388849 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":307,"threads_started":1}
I20260923 20:58:37.391767 20085 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35673
I20260923 20:58:37.392145 20085 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:37.393102 20085 heartbeater.cc:511] Master 127.0.97.62:35673 requested a full tablet report, sending...
I20260923 20:58:37.394991 19762 ts_manager.cc:194] Registered new tserver with Master: b1003d805eb74a5aa45ff545cdc38604 (127.0.97.2:35171)
I20260923 20:58:37.396355 19762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46949
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 20:58:37.433985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:37.794529 20091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:37.794824 20091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:37.794977 20091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:37.802588 20091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:37.802716 20091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:37.802929 20091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:58:37.813611 20091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:37.815029 20091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:37.816453 20091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:37.830689 20096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:37.830773 20097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:37.831650 20091 server_base.cc:1034] running on GCE node
W20260923 20:58:37.830725 20099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:37.834678 20091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:37.835806 20091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:37.837067 20091 hybrid_clock.cc:648] HybridClock initialized: now 1790197117837004 us; error 64 us; skew 500 ppm
I20260923 20:58:37.837411 20091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:37 dist-test-slave-plcg krb5kdc[19717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:37.840783 20091 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 20:58:37.842422 20091 webserver.cc:460] Webserver started at http://127.0.97.3:38669/ using document root <none> and password file <none>
I20260923 20:58:37.843201 20091 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:37.843359 20091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:37.843699 20091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:37.846876 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "b1058e95f5844951981168be66b903ee"
format_stamp: "Formatted at 2026-09-23 20:58:37 on dist-test-slave-plcg"
server_key: "230f6ed896ca661ce08f7cad4907a9b1"
server_key_iv: "33cdf941724051f8a988bb2ae909cb0b"
server_key_version: "encryptionkey@0"
I20260923 20:58:37.847760 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b1058e95f5844951981168be66b903ee"
format_stamp: "Formatted at 2026-09-23 20:58:37 on dist-test-slave-plcg"
server_key: "230f6ed896ca661ce08f7cad4907a9b1"
server_key_iv: "33cdf941724051f8a988bb2ae909cb0b"
server_key_version: "encryptionkey@0"
I20260923 20:58:37.853543 20091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:58:37.857486 20106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:37.859193 20091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:37.859403 20091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "b1058e95f5844951981168be66b903ee"
format_stamp: "Formatted at 2026-09-23 20:58:37 on dist-test-slave-plcg"
server_key: "230f6ed896ca661ce08f7cad4907a9b1"
server_key_iv: "33cdf941724051f8a988bb2ae909cb0b"
server_key_version: "encryptionkey@0"
I20260923 20:58:37.859638 20091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:37.869758 19951 heartbeater.cc:503] Master 127.0.97.62:35673 was elected leader, sending a full tablet report...
I20260923 20:58:37.891116 20091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:37.895641 20091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:37.895907 20091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:37.896885 20091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:37.898310 20091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:37.898389 20091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:37.898483 20091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:37.898528 20091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:37.933110 20091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44045
I20260923 20:58:37.933157 20218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44045 every 8 connection(s)
I20260923 20:58:37.934490 20091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:58:37.940872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 20091
I20260923 20:58:37.941115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:58:37.941640 388 external_mini_cluster.cc:1428] Setting key 092544f2bce04c36caa55687632d839b
Sep 23 20:58:37 dist-test-slave-plcg krb5kdc[19717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:58:37.957154 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:37.937195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46669 (local address 127.0.97.62:35673)
0923 20:58:37.937703 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:37.937709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:37.938801 (+ 1092us) server_negotiation.cc:407] Connection header received
0923 20:58:37.940636 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:37.940641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:37.940749 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:37.940873 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:37.942797 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.943987 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.946055 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.946479 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.946603 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:37.950400 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:37.950406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:37.950410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:37.950463 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:37.953382 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:37.954344 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:37.954350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:37.954354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:37.954512 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:37.954973 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:37.954977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:37.954979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:37.955377 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:37.955546 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:37.956518 (+ 972us) server_negotiation.cc:299] Negotiation successful
0923 20:58:37.956760 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":209,"threads_started":1}
I20260923 20:58:37.958349 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:37.937867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35673 (local address 127.0.97.3:46669)
0923 20:58:37.938620 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:37.938670 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:37.940332 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:37.941149 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:37.941159 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:37.941791 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:37.942604 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:37.942626 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.944161 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.944167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:37.945898 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:37.945913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.946701 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.946706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:37.946921 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:37.947638 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:58:37.947661 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:37.950117 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:37.953556 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:37.953563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:37.953578 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:37.954188 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:37.954624 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:58:37.954628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:58:37.954630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:58:37.954823 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:58:37.955528 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:37.955535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:37.956017 (+ 482us) client_negotiation.cc:769] Sending connection context
0923 20:58:37.956377 (+ 360us) client_negotiation.cc:240] Negotiation successful
0923 20:58:37.956970 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":280,"threads_started":1}
I20260923 20:58:37.959800 20219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35673
I20260923 20:58:37.960140 20219 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:37.960995 20219 heartbeater.cc:511] Master 127.0.97.62:35673 requested a full tablet report, sending...
I20260923 20:58:37.962682 19762 ts_manager.cc:194] Registered new tserver with Master: b1058e95f5844951981168be66b903ee (127.0.97.3:44045)
I20260923 20:58:37.963637 19762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46669
I20260923 20:58:37.970379 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:58:37.987702 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:37.974734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.0.97.62:35673)
0923 20:58:37.974967 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:37.974976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:37.975360 (+ 384us) server_negotiation.cc:407] Connection header received
0923 20:58:37.975483 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:37.975487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:37.975584 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:37.975675 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:37.977139 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.977968 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.979276 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:37.979665 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:37.979768 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:37.982274 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:37.982279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:37.982283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:37.982320 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:37.985074 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:37.985741 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:37.985745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:37.985747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:37.985946 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:37.986274 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:37.986277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:37.986279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:37.986614 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:37.986798 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:37.987141 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 20:58:37.987338 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:58:37.991704 19762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:58:37.999009 19762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:58:38.036514 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.012508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44045 (local address 127.0.0.1:48614)
0923 20:58:38.028541 (+ 16033us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.028565 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:38.028755 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:38.029869 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:38.029878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:38.029910 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:38.030416 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.030427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.031851 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.031859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.034073 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.034088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.035469 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.035475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.035675 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.035744 (+ 69us) client_negotiation.cc:769] Sending connection context
0923 20:58:38.035897 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 20:58:38.036122 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15897,"threads_started":1}
I20260923 20:58:38.036911 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.012804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.97.3:44045)
0923 20:58:38.028291 (+ 15487us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.028305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.028695 (+ 390us) server_negotiation.cc:407] Connection header received
0923 20:58:38.028946 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.028955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.029249 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.029488 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:38.030614 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.031699 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.034250 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.035241 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.035408 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.035676 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.036364 (+ 688us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.036571 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15235,"thread_start_us":15527,"threads_started":1}
I20260923 20:58:38.043841 20154 tablet_service.cc:1467] Processing CreateTablet for tablet fd6009690339444aa3b931a4a0699c3d (DEFAULT_TABLE table=test-table [id=8d72a0c5614b4152bdaa69f794aabc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:38.045454 20154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6009690339444aa3b931a4a0699c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:38.045792 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.016071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45122)
0923 20:58:38.037297 (+ 21226us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.037317 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:38.037439 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:38.038920 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:38.038925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:38.038955 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:38.039174 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.039189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.041089 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.041099 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.042836 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.042857 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.044824 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.044830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.045068 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.045103 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:58:38.045217 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 20:58:38.045362 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20843,"thread_start_us":4232,"threads_started":1}
I20260923 20:58:38.045862 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.016991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.0.97.1:42477)
0923 20:58:38.038331 (+ 21340us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.038339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.038366 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:58:38.038451 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.038457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.038686 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.038936 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:38.039397 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.040923 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.043014 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.044619 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.044778 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.045010 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.045285 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.045488 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21184,"thread_start_us":21029,"threads_started":1}
I20260923 20:58:38.048074 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.012428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35171 (local address 127.0.0.1:46356)
0923 20:58:38.020940 (+ 8512us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.021018 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:38.021307 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:38.040504 (+ 19197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:38.040510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:38.040545 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:38.041038 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.041054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.043110 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.043117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.045169 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.045211 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.047203 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.047210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.047409 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.047446 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:58:38.047542 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 20:58:38.047706 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8100,"thread_start_us":16442,"threads_started":1}
I20260923 20:58:38.048866 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.012514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.97.2:35171)
0923 20:58:38.039867 (+ 27353us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.039876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.039904 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:58:38.040004 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.040011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.040257 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.040474 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:38.041281 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.042901 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.045403 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.046960 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.047110 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.047290 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.048336 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.048526 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27168,"thread_start_us":27052,"threads_started":1}
I20260923 20:58:38.050625 19886 tablet_service.cc:1467] Processing CreateTablet for tablet fd6009690339444aa3b931a4a0699c3d (DEFAULT_TABLE table=test-table [id=8d72a0c5614b4152bdaa69f794aabc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:38.052305 19886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6009690339444aa3b931a4a0699c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:38.052611 20020 tablet_service.cc:1467] Processing CreateTablet for tablet fd6009690339444aa3b931a4a0699c3d (DEFAULT_TABLE table=test-table [id=8d72a0c5614b4152bdaa69f794aabc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:38.054108 20020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6009690339444aa3b931a4a0699c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:38.059839 20238 tablet_bootstrap.cc:492] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Bootstrap starting.
I20260923 20:58:38.065508 20238 tablet_bootstrap.cc:654] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:38.067005 20238 log.cc:824] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:38.067353 20239 tablet_bootstrap.cc:492] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: Bootstrap starting.
I20260923 20:58:38.070448 20240 tablet_bootstrap.cc:492] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Bootstrap starting.
I20260923 20:58:38.070778 20238 tablet_bootstrap.cc:492] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: No bootstrap required, opened a new log
I20260923 20:58:38.071072 20238 ts_tablet_manager.cc:1397] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260923 20:58:38.071574 20239 tablet_bootstrap.cc:654] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:38.072952 20239 log.cc:824] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:38.074361 20240 tablet_bootstrap.cc:654] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:38.075706 20240 log.cc:824] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:38.076573 20239 tablet_bootstrap.cc:492] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: No bootstrap required, opened a new log
I20260923 20:58:38.076887 20239 ts_tablet_manager.cc:1397] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 20:58:38.079231 20240 tablet_bootstrap.cc:492] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: No bootstrap required, opened a new log
I20260923 20:58:38.079541 20240 ts_tablet_manager.cc:1397] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:58:38.080084 20238 raft_consensus.cc:348] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.080868 20238 raft_consensus.cc:374] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:38.081046 20238 raft_consensus.cc:729] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1058e95f5844951981168be66b903ee, State: Initialized, Role: FOLLOWER
I20260923 20:58:38.081863 20238 consensus_queue.cc:260] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.083081 20219 heartbeater.cc:503] Master 127.0.97.62:35673 was elected leader, sending a full tablet report...
I20260923 20:58:38.083719 20238 ts_tablet_manager.cc:1428] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Time spent starting tablet: real 0.013s user 0.003s sys 0.009s
I20260923 20:58:38.088979 20239 raft_consensus.cc:348] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.089007 20240 raft_consensus.cc:348] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.089495 20239 raft_consensus.cc:374] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:38.089601 20239 raft_consensus.cc:729] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38e7207da3d4ebaa3772e04205748f7, State: Initialized, Role: FOLLOWER
I20260923 20:58:38.089620 20240 raft_consensus.cc:374] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:38.089762 20240 raft_consensus.cc:729] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1003d805eb74a5aa45ff545cdc38604, State: Initialized, Role: FOLLOWER
I20260923 20:58:38.090392 20239 consensus_queue.cc:260] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.090497 20240 consensus_queue.cc:260] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.091698 20085 heartbeater.cc:503] Master 127.0.97.62:35673 was elected leader, sending a full tablet report...
I20260923 20:58:38.092485 20240 ts_tablet_manager.cc:1428] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260923 20:58:38.092491 20239 ts_tablet_manager.cc:1428] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260923 20:58:38.119477 20086 tablet.cc:2369] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:58:38.186200 20220 tablet.cc:2369] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:38.216960 20245 raft_consensus.cc:482] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:58:38.217250 20245 raft_consensus.cc:504] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.219022 20245 leader_election.cc:288] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1003d805eb74a5aa45ff545cdc38604 (127.0.97.2:35171), b1058e95f5844951981168be66b903ee (127.0.97.3:44045)
I20260923 20:58:38.226717 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.219558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34809 (local address 127.0.97.3:44045)
0923 20:58:38.219777 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.219786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.220695 (+ 909us) server_negotiation.cc:407] Connection header received
0923 20:58:38.220903 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.220912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.220999 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.221105 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:38.221784 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.222849 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.224820 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.225696 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.225886 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.226012 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.226311 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.226446 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:58:38.226823 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.219867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46261 (local address 127.0.97.2:35171)
0923 20:58:38.220123 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.220129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.220649 (+ 520us) server_negotiation.cc:407] Connection header received
0923 20:58:38.220885 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.220890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.220953 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.221053 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:38.221719 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.222681 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.224765 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.225810 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.226012 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.226119 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.226413 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.226558 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:58:38.226931 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.219577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35171 (local address 127.0.97.1:46261)
0923 20:58:38.220504 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.220533 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:38.220726 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:38.221084 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:38.221089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:38.221115 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:38.221560 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.221572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.222821 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.222827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.224578 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.224593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.226089 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.226096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.226253 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.226276 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:58:38.226386 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:58:38.226522 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":5,"thread_start_us":465,"threads_started":1}
I20260923 20:58:38.227281 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.219577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44045 (local address 127.0.97.1:34809)
0923 20:58:38.220567 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.220582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:38.220749 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:38.221123 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:38.221128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:38.221150 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:38.221588 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.221597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.223006 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.223012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.224646 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:38.224659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.225954 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.225961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:38.226136 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.226162 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:58:38.226281 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 20:58:38.226426 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":121,"thread_start_us":570,"threads_started":1}
I20260923 20:58:38.227967 20040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6009690339444aa3b931a4a0699c3d" candidate_uuid: "c38e7207da3d4ebaa3772e04205748f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1003d805eb74a5aa45ff545cdc38604" is_pre_election: true
I20260923 20:58:38.228248 20174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6009690339444aa3b931a4a0699c3d" candidate_uuid: "c38e7207da3d4ebaa3772e04205748f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1058e95f5844951981168be66b903ee" is_pre_election: true
I20260923 20:58:38.228488 20040 raft_consensus.cc:2454] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38e7207da3d4ebaa3772e04205748f7 in term 0.
I20260923 20:58:38.228778 20174 raft_consensus.cc:2454] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38e7207da3d4ebaa3772e04205748f7 in term 0.
I20260923 20:58:38.229238 19842 leader_election.cc:302] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1003d805eb74a5aa45ff545cdc38604, c38e7207da3d4ebaa3772e04205748f7; no voters:
I20260923 20:58:38.229681 20245 raft_consensus.cc:2781] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:58:38.229873 20245 raft_consensus.cc:482] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:58:38.230005 20245 raft_consensus.cc:3037] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:38.232257 20245 raft_consensus.cc:504] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.232928 20245 leader_election.cc:288] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [CANDIDATE]: Term 1 election: Requested vote from peers b1003d805eb74a5aa45ff545cdc38604 (127.0.97.2:35171), b1058e95f5844951981168be66b903ee (127.0.97.3:44045)
I20260923 20:58:38.233441 20040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6009690339444aa3b931a4a0699c3d" candidate_uuid: "c38e7207da3d4ebaa3772e04205748f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1003d805eb74a5aa45ff545cdc38604"
I20260923 20:58:38.233469 20174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6009690339444aa3b931a4a0699c3d" candidate_uuid: "c38e7207da3d4ebaa3772e04205748f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1058e95f5844951981168be66b903ee"
I20260923 20:58:38.233676 20040 raft_consensus.cc:3037] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:38.233678 20174 raft_consensus.cc:3037] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:38.235968 20040 raft_consensus.cc:2454] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38e7207da3d4ebaa3772e04205748f7 in term 1.
I20260923 20:58:38.235998 20174 raft_consensus.cc:2454] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38e7207da3d4ebaa3772e04205748f7 in term 1.
I20260923 20:58:38.236516 19839 leader_election.cc:302] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1058e95f5844951981168be66b903ee, c38e7207da3d4ebaa3772e04205748f7; no voters:
I20260923 20:58:38.236843 20245 raft_consensus.cc:2781] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:38.237247 20245 raft_consensus.cc:686] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 1 LEADER]: Becoming Leader. State: Replica: c38e7207da3d4ebaa3772e04205748f7, State: Running, Role: LEADER
I20260923 20:58:38.237766 20245 consensus_queue.cc:237] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } }
I20260923 20:58:38.243459 19761 catalog_manager.cc:5495] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c38e7207da3d4ebaa3772e04205748f7 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c38e7207da3d4ebaa3772e04205748f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c38e7207da3d4ebaa3772e04205748f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42477 } health_report { overall_health: HEALTHY } } }
I20260923 20:58:38.283660 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.276506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.0.97.1:42477)
0923 20:58:38.276773 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.276780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.276804 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:58:38.277099 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.277104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.277197 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.277393 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:58:38.277869 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.278666 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.280714 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.281442 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.281609 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.282364 (+ 755us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:58:38.282993 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:58:38.283165 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.283269 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.283360 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 20:58:38.292654 20040 raft_consensus.cc:1261] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 1 FOLLOWER]: Refusing update from remote peer c38e7207da3d4ebaa3772e04205748f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:38.292675 20174 raft_consensus.cc:1261] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 1 FOLLOWER]: Refusing update from remote peer c38e7207da3d4ebaa3772e04205748f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:38.293752 20249 consensus_queue.cc:1035] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1003d805eb74a5aa45ff545cdc38604" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:38.294226 20245 consensus_queue.cc:1035] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1058e95f5844951981168be66b903ee" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:38.325285 19761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:38.325616 19761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:38.329468 19761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8548456554a4415fbf9ea13a16062e4e: Sending DeleteTablet for 3 replicas of tablet fd6009690339444aa3b931a4a0699c3d
I20260923 20:58:38.330763 20154 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6009690339444aa3b931a4a0699c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48614
I20260923 20:58:38.330767 19886 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6009690339444aa3b931a4a0699c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45122
I20260923 20:58:38.331560 20020 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6009690339444aa3b931a4a0699c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260923 20:58:38.334424 20264 tablet_replica.cc:330] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: stopping tablet replica
I20260923 20:58:38.335008 20264 raft_consensus.cc:2229] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:38.335721 20264 raft_consensus.cc:2258] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:38.336206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19823
I20260923 20:58:38.346045 20264 ts_tablet_manager.cc:1905] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:58:38 dist-test-slave-plcg krb5kdc[19717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 20:58:38.355855 20262 tablet_replica.cc:330] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: stopping tablet replica
I20260923 20:58:38.357161 20262 raft_consensus.cc:2229] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:58:38.360816 20262 raft_consensus.cc:2258] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:38.361802 20263 tablet_replica.cc:330] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: stopping tablet replica
I20260923 20:58:38.362356 20263 raft_consensus.cc:2229] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:38.362943 20263 raft_consensus.cc:2258] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:38.363678 20264 ts_tablet_manager.cc:1918] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:38.363950 20264 log.cc:1196] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/fd6009690339444aa3b931a4a0699c3d
I20260923 20:58:38.364688 20264 ts_tablet_manager.cc:1939] T fd6009690339444aa3b931a4a0699c3d P b1003d805eb74a5aa45ff545cdc38604: Deleting consensus metadata
I20260923 20:58:38.364912 20262 ts_tablet_manager.cc:1905] T fd6009690339444aa3b931a4a0699c3d P c38e7207da3d4ebaa3772e04205748f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:38.366036 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.337521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.0.97.1:42477)
0923 20:58:38.337793 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.337803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.338421 (+ 618us) server_negotiation.cc:407] Connection header received
0923 20:58:38.339242 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.339252 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.339375 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.339474 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:38.341796 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.343443 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.346145 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.347153 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.347285 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.355154 (+ 7869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:38.355178 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:38.355206 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:38.355288 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:38.360621 (+ 5333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:38.362491 (+ 1870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:38.362504 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:38.362510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:38.362690 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:38.363208 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:38.363213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:38.363217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:38.363702 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:38.364057 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.365331 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.365597 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:58:38.368001 20263 ts_tablet_manager.cc:1905] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:38.370249 19946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:38.372820 19749 catalog_manager.cc:4841] TS b1003d805eb74a5aa45ff545cdc38604 (127.0.97.2:35171): tablet fd6009690339444aa3b931a4a0699c3d (table test-table [id=8d72a0c5614b4152bdaa69f794aabc9b]) successfully deleted
I20260923 20:58:38.376626 20263 ts_tablet_manager.cc:1918] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:38.376818 20263 log.cc:1196] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/fd6009690339444aa3b931a4a0699c3d
I20260923 20:58:38.377357 20263 ts_tablet_manager.cc:1939] T fd6009690339444aa3b931a4a0699c3d P b1058e95f5844951981168be66b903ee: Deleting consensus metadata
I20260923 20:58:38.379379 19746 catalog_manager.cc:4841] TS b1058e95f5844951981168be66b903ee (127.0.97.3:44045): tablet fd6009690339444aa3b931a4a0699c3d (table test-table [id=8d72a0c5614b4152bdaa69f794aabc9b]) successfully deleted
I20260923 20:58:38.472812 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19823
W20260923 20:58:38.491475 19747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42477 (error 108)
I20260923 20:58:38.491899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19957
I20260923 20:58:38.493316 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.492727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45144)
0923 20:58:38.492991 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.493146 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260923 20:58:38.493594 19747 catalog_manager.cc:4584] TS c38e7207da3d4ebaa3772e04205748f7 (127.0.97.1:42477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd6009690339444aa3b931a4a0699c3d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Sep 23 20:58:38 dist-test-slave-plcg krb5kdc[19717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 20:58:38.507083 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.492728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.97.2:35171)
0923 20:58:38.492934 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.492940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.492969 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:58:38.493126 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.493131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.493205 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.493290 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:38.494642 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.495352 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.496577 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.496951 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.497050 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.501298 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:38.501313 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:38.501329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:38.501371 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:38.504387 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:38.505137 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:38.505145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:38.505147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:38.505278 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:38.505628 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:38.505632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:38.505634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:38.506090 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:38.506350 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.506605 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.506821 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:58:38.507750 20080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:38.547477 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.546925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45154)
0923 20:58:38.547154 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.547266 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:58:38.548682 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.548271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45158)
0923 20:58:38.548435 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.548527 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 20:58:38.599782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19957
I20260923 20:58:38.609115 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.608620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45168)
0923 20:58:38.608847 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.608939 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:58:38.610184 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.609805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45178)
0923 20:58:38.609996 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.610073 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 20:58:38.618803 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20091
Sep 23 20:58:38 dist-test-slave-plcg krb5kdc[19717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 20:58:38.634734 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.619586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.0.97.3:44045)
0923 20:58:38.619781 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:38.619785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:38.619803 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:58:38.619863 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:38.619867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:38.619933 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:38.620041 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:38.621374 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.622237 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.623520 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:38.623963 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:38.624069 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:38.629087 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:38.629104 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:38.629121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:38.629162 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:38.631994 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:38.632799 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:38.632807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:38.632809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:38.632967 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:38.633306 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:38.633309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:38.633311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:38.633701 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:38.634010 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:38.634266 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 20:58:38.634459 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:58:38.635376 20214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:38.690994 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.690477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45186)
0923 20:58:38.690719 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.690815 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 20:58:38.692201 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:38.691784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42477 (local address 127.0.0.1:45190)
0923 20:58:38.691983 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:38.692065 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 20:58:38.726876 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 20091
I20260923 20:58:38.744004 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 19730
I20260923 20:58:38.745151 19792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:38.862565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 19730
2026-09-23T20:58:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4550 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T20:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:38Z Disabled control of system clock
I20260923 20:58:38.942739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44231
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:58:39.290210 20274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:39.290525 20274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:39.290657 20274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:39.298411 20274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:39.298542 20274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:39.298631 20274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:39.298739 20274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:39.298815 20274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:39.308653 20274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:39.310099 20274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:39.311968 20274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:39.324678 20279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:39.324781 20282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:39.324959 20280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:39.325979 20274 server_base.cc:1034] running on GCE node
I20260923 20:58:39.326777 20274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:39.328296 20274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:39.329509 20274 hybrid_clock.cc:648] HybridClock initialized: now 1790197119329455 us; error 44 us; skew 500 ppm
I20260923 20:58:39.329983 20274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:39.337872 20274 webserver.cc:460] Webserver started at http://127.0.97.62:37413/ using document root <none> and password file <none>
I20260923 20:58:39.338691 20274 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:39.338821 20274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:39.339160 20274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:39.342355 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb"
format_stamp: "Formatted at 2026-09-23 20:58:39 on dist-test-slave-plcg"
server_key: "5112ac9202025585143d71c9a4ac9ccd"
server_key_iv: "c8cc995e441c8c68cd4834b2e71f142e"
server_key_version: "encryptionkey@0"
I20260923 20:58:39.343247 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb"
format_stamp: "Formatted at 2026-09-23 20:58:39 on dist-test-slave-plcg"
server_key: "5112ac9202025585143d71c9a4ac9ccd"
server_key_iv: "c8cc995e441c8c68cd4834b2e71f142e"
server_key_version: "encryptionkey@0"
I20260923 20:58:39.348938 20274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:58:39.352843 20288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:39.354410 20274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:39.354635 20274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb"
format_stamp: "Formatted at 2026-09-23 20:58:39 on dist-test-slave-plcg"
server_key: "5112ac9202025585143d71c9a4ac9ccd"
server_key_iv: "c8cc995e441c8c68cd4834b2e71f142e"
server_key_version: "encryptionkey@0"
I20260923 20:58:39.354877 20274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:39.372813 20274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:39.373729 20274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:39.374078 20274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:39.393539 20274 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44231
I20260923 20:58:39.393559 20339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44231 every 8 connection(s)
I20260923 20:58:39.395031 20274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:39.398954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 20274
I20260923 20:58:39.398977 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:39.399142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:39.399583 388 external_mini_cluster.cc:1428] Setting key 7b3886b828287faf3e175be38e86b6e7
I20260923 20:58:39.415431 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb: Bootstrap starting.
I20260923 20:58:39.415609 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:39.401539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.0.97.62:44231)
0923 20:58:39.402887 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:39.402905 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:39.402958 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:58:39.404689 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:39.404728 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:39.405443 (+ 715us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:39.405944 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:39.406560 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:39.408619 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:39.410227 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:39.411320 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:39.411510 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:39.412643 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:39.412659 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:39.412681 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:39.413465 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:39.413565 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:39.413590 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:39.413714 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:39.413900 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 20:58:39.414260 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":473,"threads_started":1}
I20260923 20:58:39.418982 20340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:39.419996 20340 log.cc:824] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:39.422887 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb: No bootstrap required, opened a new log
I20260923 20:58:39.431133 20340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260923 20:58:39.431538 20340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:39.431623 20340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8057c3f0f4a54c06a8cdb6ed124f9adb, State: Initialized, Role: FOLLOWER
I20260923 20:58:39.432140 20340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260923 20:58:39.432346 20340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:39.432449 20340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:39.432579 20340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:39.434930 20340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260923 20:58:39.435537 20340 leader_election.cc:302] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8057c3f0f4a54c06a8cdb6ed124f9adb; no voters:
I20260923 20:58:39.436268 20340 leader_election.cc:288] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:39.436403 20345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:39.437180 20345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [term 1 LEADER]: Becoming Leader. State: Replica: 8057c3f0f4a54c06a8cdb6ed124f9adb, State: Running, Role: LEADER
I20260923 20:58:39.437727 20345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260923 20:58:39.438568 20340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:39.440641 20347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8057c3f0f4a54c06a8cdb6ed124f9adb. Latest consensus state: current_term: 1 leader_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } } }
I20260923 20:58:39.441159 20347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:39.440769 20346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8057c3f0f4a54c06a8cdb6ed124f9adb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } } }
I20260923 20:58:39.441645 20346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:39.443655 20353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:39.448774 20353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:39.457921 20353 catalog_manager.cc:1347] Generated new cluster ID: 5204bd48f8f049f687e5d0ecd1ad2129
I20260923 20:58:39.458072 20353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:39.473970 20353 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:39.475931 20353 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:39.484417 20353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb: Generated new TSK 0
I20260923 20:58:39.485620 20353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:39.492126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--builtin_ntp_servers=127.0.97.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:58:39.852577 20364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:39.852941 20364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:39.853125 20364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:39.860816 20364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:39.860980 20364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:39.861173 20364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 20:58:39.870720 20364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:39.872136 20364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:39.874051 20364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:39.889479 20369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:39.891044 20372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:39.891769 20364 server_base.cc:1034] running on GCE node
W20260923 20:58:39.891886 20370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:39.892655 20364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:39.893935 20364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:39.895190 20364 hybrid_clock.cc:648] HybridClock initialized: now 1790197119895125 us; error 55 us; skew 500 ppm
I20260923 20:58:39.895548 20364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:39.898089 20364 webserver.cc:460] Webserver started at http://127.0.97.1:38685/ using document root <none> and password file <none>
I20260923 20:58:39.898857 20364 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:39.899001 20364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:39.899349 20364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:39.902539 20364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data/instance:
uuid: "d0015fe049a2451abc4839084a1bac49"
format_stamp: "Formatted at 2026-09-23 20:58:39 on dist-test-slave-plcg"
server_key: "4829c0acd08129a4a0acdc2dba1f2a9c"
server_key_iv: "a22d556d53d3e64b3cb962d8112dadef"
server_key_version: "encryptionkey@0"
I20260923 20:58:39.903388 20364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d0015fe049a2451abc4839084a1bac49"
format_stamp: "Formatted at 2026-09-23 20:58:39 on dist-test-slave-plcg"
server_key: "4829c0acd08129a4a0acdc2dba1f2a9c"
server_key_iv: "a22d556d53d3e64b3cb962d8112dadef"
server_key_version: "encryptionkey@0"
I20260923 20:58:39.908839 20364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:58:39.912817 20378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:39.914342 20364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:39.914539 20364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/wal
uuid: "d0015fe049a2451abc4839084a1bac49"
format_stamp: "Formatted at 2026-09-23 20:58:39 on dist-test-slave-plcg"
server_key: "4829c0acd08129a4a0acdc2dba1f2a9c"
server_key_iv: "a22d556d53d3e64b3cb962d8112dadef"
server_key_version: "encryptionkey@0"
I20260923 20:58:39.914795 20364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:39.939190 20364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:39.940148 20364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:39.940442 20364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:39.941465 20364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:39.943216 20364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:39.943347 20364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:39.943475 20364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:39.943589 20364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:39.977988 20364 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40331
I20260923 20:58:39.978081 20490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40331 every 8 connection(s)
I20260923 20:58:39.979350 20364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/data/info.pb
I20260923 20:58:39.981552 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 20364
I20260923 20:58:39.981735 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-0/wal/instance
I20260923 20:58:39.982272 388 external_mini_cluster.cc:1428] Setting key 6203ea86faab038e8a86f607903500b6
I20260923 20:58:39.985747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--builtin_ntp_servers=127.0.97.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:58:39.996387 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:39.982150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41899 (local address 127.0.97.62:44231)
0923 20:58:39.982704 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:39.982711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:39.984247 (+ 1536us) server_negotiation.cc:407] Connection header received
0923 20:58:39.987217 (+ 2970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:39.987223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:39.987351 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:39.987477 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:39.989114 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:39.989993 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:39.992742 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:39.993414 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:39.993555 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:39.995433 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:39.995438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:39.995441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:39.995462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:39.995509 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:39.995522 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:39.995631 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:39.995882 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 20:58:39.995998 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
I20260923 20:58:39.997511 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:39.982831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44231 (local address 127.0.97.1:41899)
0923 20:58:39.984062 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:39.984114 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:39.986802 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:39.987798 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:39.987814 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:39.987954 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:39.988861 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:39.988893 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:39.990191 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:39.990200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:39.992523 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:39.992547 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:39.993696 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:39.993705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:39.994114 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:39.995051 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:39.995054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:39.995099 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:39.995190 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:39.995632 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:39.995646 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:39.995657 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:58:39.995868 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 20:58:39.996349 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":433,"threads_started":1}
I20260923 20:58:39.999387 20491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44231
I20260923 20:58:39.999773 20491 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:40.000658 20491 heartbeater.cc:511] Master 127.0.97.62:44231 requested a full tablet report, sending...
I20260923 20:58:40.003363 20305 ts_manager.cc:194] Registered new tserver with Master: d0015fe049a2451abc4839084a1bac49 (127.0.97.1:40331)
I20260923 20:58:40.005690 20305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:41899
W20260923 20:58:40.330304 20495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:40.330657 20495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:40.330812 20495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:40.338485 20495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:40.338637 20495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:40.338832 20495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 20:58:40.348357 20495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:40.349640 20495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:40.351452 20495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:40.367687 20501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:40.367636 20503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:40.368191 20500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:40.369611 20495 server_base.cc:1034] running on GCE node
I20260923 20:58:40.370338 20495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:40.371382 20495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:40.372634 20495 hybrid_clock.cc:648] HybridClock initialized: now 1790197120372586 us; error 42 us; skew 500 ppm
I20260923 20:58:40.372988 20495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:40.375495 20495 webserver.cc:460] Webserver started at http://127.0.97.2:35387/ using document root <none> and password file <none>
I20260923 20:58:40.376286 20495 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:40.376427 20495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:40.376772 20495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:40.379879 20495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data/instance:
uuid: "b771d5b64e4245f8bb1c090f1c4658c0"
format_stamp: "Formatted at 2026-09-23 20:58:40 on dist-test-slave-plcg"
server_key: "69661f99c8c07dc6def91caf64f42c1d"
server_key_iv: "f77f872ca61b1ad6a5f1444fa658e784"
server_key_version: "encryptionkey@0"
I20260923 20:58:40.380764 20495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b771d5b64e4245f8bb1c090f1c4658c0"
format_stamp: "Formatted at 2026-09-23 20:58:40 on dist-test-slave-plcg"
server_key: "69661f99c8c07dc6def91caf64f42c1d"
server_key_iv: "f77f872ca61b1ad6a5f1444fa658e784"
server_key_version: "encryptionkey@0"
I20260923 20:58:40.386411 20495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:58:40.390331 20509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:40.391736 20495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:40.391937 20495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal
uuid: "b771d5b64e4245f8bb1c090f1c4658c0"
format_stamp: "Formatted at 2026-09-23 20:58:40 on dist-test-slave-plcg"
server_key: "69661f99c8c07dc6def91caf64f42c1d"
server_key_iv: "f77f872ca61b1ad6a5f1444fa658e784"
server_key_version: "encryptionkey@0"
I20260923 20:58:40.392191 20495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:40.409198 20495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:40.410326 20495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:40.410648 20495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:40.411821 20495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:40.413725 20495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:40.413878 20495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:40.413988 20495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:40.414060 20495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:40.448698 20495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38475
I20260923 20:58:40.448755 20621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38475 every 8 connection(s)
I20260923 20:58:40.450109 20495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/data/info.pb
I20260923 20:58:40.456755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 20495
I20260923 20:58:40.456933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal/instance
I20260923 20:58:40.457409 388 external_mini_cluster.cc:1428] Setting key 434c35b3e2ea57ecf4d336854ede0637
I20260923 20:58:40.460359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--builtin_ntp_servers=127.0.97.20:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:58:40.463953 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.452879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36869 (local address 127.0.97.62:44231)
0923 20:58:40.453080 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:40.453085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:40.454519 (+ 1434us) server_negotiation.cc:407] Connection header received
0923 20:58:40.456431 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:40.456439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:40.456556 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:40.456677 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:40.458044 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.458956 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.460895 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.461313 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.461458 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:40.463075 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:40.463080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:40.463083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:40.463104 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:40.463137 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:40.463149 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:40.463253 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:40.463510 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 20:58:40.463622 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:58:40.465076 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.453317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44231 (local address 127.0.97.2:36869)
0923 20:58:40.454342 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:40.454381 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:40.456114 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:40.456961 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:40.456991 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:40.457126 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:40.457802 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:40.457872 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.459136 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.459143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:40.460724 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:40.460740 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.461579 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.461589 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:40.461901 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:40.462692 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:40.462694 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:40.462740 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:40.462837 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:40.463254 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:40.463268 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:40.463279 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:58:40.463470 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 20:58:40.463927 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":303,"threads_started":1}
I20260923 20:58:40.466842 20622 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44231
I20260923 20:58:40.467190 20622 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:40.467986 20622 heartbeater.cc:511] Master 127.0.97.62:44231 requested a full tablet report, sending...
I20260923 20:58:40.469648 20305 ts_manager.cc:194] Registered new tserver with Master: b771d5b64e4245f8bb1c090f1c4658c0 (127.0.97.2:38475)
I20260923 20:58:40.470534 20305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36869
W20260923 20:58:40.806360 20625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:40.806712 20625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:40.806883 20625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:40.814450 20625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:40.814603 20625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:40.814783 20625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 20:58:40.824491 20625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:40.825976 20625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:40.827826 20625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:40.841620 20630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:40.841646 20631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:40.841619 20633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:40.847651 20625 server_base.cc:1034] running on GCE node
I20260923 20:58:40.848397 20625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:40.849521 20625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:40.850780 20625 hybrid_clock.cc:648] HybridClock initialized: now 1790197120850705 us; error 79 us; skew 500 ppm
I20260923 20:58:40.851140 20625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:58:40.853497 20625 webserver.cc:460] Webserver started at http://127.0.97.3:36447/ using document root <none> and password file <none>
I20260923 20:58:40.854350 20625 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:40.854493 20625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:40.854825 20625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:40.858052 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data/instance:
uuid: "473400b17f614d9cb54461351ea92885"
format_stamp: "Formatted at 2026-09-23 20:58:40 on dist-test-slave-plcg"
server_key: "a8cd7ca633b1dee55ec2d07bac62f502"
server_key_iv: "4c5d04b9ab1cbb4b157bb3ee02345be0"
server_key_version: "encryptionkey@0"
I20260923 20:58:40.859347 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal/instance:
uuid: "473400b17f614d9cb54461351ea92885"
format_stamp: "Formatted at 2026-09-23 20:58:40 on dist-test-slave-plcg"
server_key: "a8cd7ca633b1dee55ec2d07bac62f502"
server_key_iv: "4c5d04b9ab1cbb4b157bb3ee02345be0"
server_key_version: "encryptionkey@0"
I20260923 20:58:40.865015 20625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:58:40.868846 20639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:40.870510 20625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:58:40.870714 20625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal
uuid: "473400b17f614d9cb54461351ea92885"
format_stamp: "Formatted at 2026-09-23 20:58:40 on dist-test-slave-plcg"
server_key: "a8cd7ca633b1dee55ec2d07bac62f502"
server_key_iv: "4c5d04b9ab1cbb4b157bb3ee02345be0"
server_key_version: "encryptionkey@0"
I20260923 20:58:40.870968 20625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:40.890362 20625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:40.891239 20625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:40.891547 20625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:40.892555 20625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:58:40.894152 20625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:58:40.894279 20625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:40.894411 20625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:58:40.894510 20625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:40.929095 20625 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46297
I20260923 20:58:40.929144 20751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46297 every 8 connection(s)
I20260923 20:58:40.930619 20625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/data/info.pb
I20260923 20:58:40.938127 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 20625
I20260923 20:58:40.938345 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal/instance
I20260923 20:58:40.938796 388 external_mini_cluster.cc:1428] Setting key 82e7568c199bf4cf74e8fa518648df28
I20260923 20:58:40.944711 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.933275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43057 (local address 127.0.97.62:44231)
0923 20:58:40.933465 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:40.933469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:40.935012 (+ 1543us) server_negotiation.cc:407] Connection header received
0923 20:58:40.936916 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:40.936923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:40.937018 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:40.937138 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:40.938349 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.939289 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.941560 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.942097 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.942201 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:40.943606 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:40.943610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:40.943614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:40.943632 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:40.943659 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:40.943669 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:40.943783 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:40.944149 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 20:58:40.944325 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:58:40.945600 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.933782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44231 (local address 127.0.97.3:43057)
0923 20:58:40.934832 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:40.934875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:40.936613 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:40.937382 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:40.937392 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:40.937485 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:58:40.938163 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:40.938182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.939500 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.939506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:40.941351 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:40.941366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.942301 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.942306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:40.942540 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:40.943280 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:58:40.943283 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:58:40.943326 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:58:40.943418 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:58:40.943773 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:58:40.943781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:58:40.943787 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:58:40.943947 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 20:58:40.944349 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":319,"threads_started":1}
I20260923 20:58:40.947220 20752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44231
I20260923 20:58:40.947553 20752 heartbeater.cc:464] Registering TS with master...
I20260923 20:58:40.948354 20752 heartbeater.cc:511] Master 127.0.97.62:44231 requested a full tablet report, sending...
I20260923 20:58:40.950115 20305 ts_manager.cc:194] Registered new tserver with Master: 473400b17f614d9cb54461351ea92885 (127.0.97.3:46297)
I20260923 20:58:40.950933 20305 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43057
I20260923 20:58:40.954288 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:58:40.963974 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.958440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.97.62:44231)
0923 20:58:40.958644 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:40.958648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:40.959022 (+ 374us) server_negotiation.cc:407] Connection header received
0923 20:58:40.959206 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:40.959210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:40.959286 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:40.959421 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:40.959853 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.960548 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.961915 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:40.962325 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:40.962428 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:40.963390 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:40.963395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:40.963398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:40.963415 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:40.963446 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:40.963457 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:40.963526 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:40.963641 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 20:58:40.963724 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:58:40.967799 20305 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:58:40.974928 20305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:58:41.009377 20491 heartbeater.cc:503] Master 127.0.97.62:44231 was elected leader, sending a full tablet report...
I20260923 20:58:41.013217 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.988295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46297 (local address 127.0.0.1:46798)
0923 20:58:41.000500 (+ 12205us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.000586 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:41.000863 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:41.002781 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:41.002794 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:41.002847 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:41.003412 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.003434 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.006966 (+ 3532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.006975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.010037 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.010063 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.012210 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.012220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.012462 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.012525 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 20:58:41.012691 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 20:58:41.012892 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11805,"thread_start_us":17531,"threads_started":1}
I20260923 20:58:41.013690 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.988720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.0.97.3:46297)
0923 20:58:41.001413 (+ 12693us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.001425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.001478 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:58:41.001620 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.001633 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.002026 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.002313 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:41.003787 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.006675 (+ 2888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.010380 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.011995 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.012163 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.012493 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.013148 (+ 655us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.013326 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12457,"thread_start_us":12312,"threads_started":1}
I20260923 20:58:41.018407 20687 tablet_service.cc:1467] Processing CreateTablet for tablet 511a17d308ff40eda3d4d6a9c817ff5d (DEFAULT_TABLE table=test-table [id=09541cfc41724e1ab79b25838d7ea201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:41.019151 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.989796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.0.97.1:40331)
0923 20:58:41.010047 (+ 20251us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.010058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.010776 (+ 718us) server_negotiation.cc:407] Connection header received
0923 20:58:41.010887 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.010895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.011218 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.011411 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:41.013088 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.014492 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.016698 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.018075 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.018252 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.018551 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.018630 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.018796 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19013,"thread_start_us":19977,"threads_started":1}
I20260923 20:58:41.019446 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.989148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40024)
0923 20:58:41.010504 (+ 21356us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.010524 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:41.010647 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:41.011651 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:41.011657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:41.011685 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:41.012886 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.012899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.014638 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.014644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.016442 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.016460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.018286 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.018292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.018465 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.018499 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:58:41.018592 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 20:58:41.018735 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21230,"mutex_wait_us":139,"thread_start_us":21069,"threads_started":1}
I20260923 20:58:41.020150 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.988742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38475 (local address 127.0.0.1:40544)
0923 20:58:41.006189 (+ 17447us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.006212 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:41.006367 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:41.011723 (+ 5356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:41.011728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:41.011761 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:41.012325 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.012342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.014366 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.014371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.016787 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.016802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.019227 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.019237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.019500 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.019535 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:58:41.019638 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:58:41.019806 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17285,"mutex_wait_us":121,"thread_start_us":11182,"threads_started":1}
I20260923 20:58:41.020231 20687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511a17d308ff40eda3d4d6a9c817ff5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:41.021011 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:40.989420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.0.97.2:38475)
0923 20:58:41.010859 (+ 21439us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.010868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.010897 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:58:41.011019 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.011026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.011259 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.011467 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:41.012536 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.014199 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.017028 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.018601 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.018742 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.018958 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.020472 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.020665 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21270,"thread_start_us":21904,"threads_started":1}
I20260923 20:58:41.023159 20426 tablet_service.cc:1467] Processing CreateTablet for tablet 511a17d308ff40eda3d4d6a9c817ff5d (DEFAULT_TABLE table=test-table [id=09541cfc41724e1ab79b25838d7ea201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:41.024480 20426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511a17d308ff40eda3d4d6a9c817ff5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:41.024570 20557 tablet_service.cc:1467] Processing CreateTablet for tablet 511a17d308ff40eda3d4d6a9c817ff5d (DEFAULT_TABLE table=test-table [id=09541cfc41724e1ab79b25838d7ea201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:58:41.025992 20557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511a17d308ff40eda3d4d6a9c817ff5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:41.036801 20771 tablet_bootstrap.cc:492] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: Bootstrap starting.
I20260923 20:58:41.040867 20771 tablet_bootstrap.cc:654] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:41.042194 20771 log.cc:824] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:41.042197 20770 tablet_bootstrap.cc:492] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Bootstrap starting.
I20260923 20:58:41.048602 20770 tablet_bootstrap.cc:654] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:41.050115 20770 log.cc:824] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:41.050478 20772 tablet_bootstrap.cc:492] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Bootstrap starting.
I20260923 20:58:41.052910 20771 tablet_bootstrap.cc:492] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: No bootstrap required, opened a new log
I20260923 20:58:41.053241 20771 ts_tablet_manager.cc:1397] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20260923 20:58:41.053406 20770 tablet_bootstrap.cc:492] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: No bootstrap required, opened a new log
I20260923 20:58:41.053695 20770 ts_tablet_manager.cc:1397] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260923 20:58:41.054098 20772 tablet_bootstrap.cc:654] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:41.055225 20772 log.cc:824] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:41.058269 20772 tablet_bootstrap.cc:492] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: No bootstrap required, opened a new log
I20260923 20:58:41.058640 20772 ts_tablet_manager.cc:1397] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:58:41.064705 20770 raft_consensus.cc:348] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.064710 20771 raft_consensus.cc:348] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.065239 20771 raft_consensus.cc:374] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:41.065239 20770 raft_consensus.cc:374] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:41.065351 20771 raft_consensus.cc:729] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0015fe049a2451abc4839084a1bac49, State: Initialized, Role: FOLLOWER
I20260923 20:58:41.065351 20770 raft_consensus.cc:729] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473400b17f614d9cb54461351ea92885, State: Initialized, Role: FOLLOWER
I20260923 20:58:41.066085 20771 consensus_queue.cc:260] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.066093 20770 consensus_queue.cc:260] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.067032 20772 raft_consensus.cc:348] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.067533 20772 raft_consensus.cc:374] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:41.067636 20772 raft_consensus.cc:729] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b771d5b64e4245f8bb1c090f1c4658c0, State: Initialized, Role: FOLLOWER
I20260923 20:58:41.067905 20752 heartbeater.cc:503] Master 127.0.97.62:44231 was elected leader, sending a full tablet report...
I20260923 20:58:41.068157 20771 ts_tablet_manager.cc:1428] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260923 20:58:41.068320 20770 ts_tablet_manager.cc:1428] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260923 20:58:41.068411 20772 consensus_queue.cc:260] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.069547 20622 heartbeater.cc:503] Master 127.0.97.62:44231 was elected leader, sending a full tablet report...
I20260923 20:58:41.070283 20772 ts_tablet_manager.cc:1428] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
W20260923 20:58:41.182514 20753 tablet.cc:2369] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:58:41.202198 20623 tablet.cc:2369] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:58:41.232021 20492 tablet.cc:2369] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:58:41.316275 20778 raft_consensus.cc:482] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:58:41.316628 20778 raft_consensus.cc:504] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.318574 20778 leader_election.cc:288] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 473400b17f614d9cb54461351ea92885 (127.0.97.3:46297), d0015fe049a2451abc4839084a1bac49 (127.0.97.1:40331)
I20260923 20:58:41.326122 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.319110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43877 (local address 127.0.97.1:40331)
0923 20:58:41.319371 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.319382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.320977 (+ 1595us) server_negotiation.cc:407] Connection header received
0923 20:58:41.321179 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.321187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.321292 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.321395 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:41.321917 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.322601 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.324369 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.325146 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.325292 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.325402 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.325711 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.325875 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:58:41.326244 20780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.318894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.97.2:43877)
0923 20:58:41.320836 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.320860 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:41.321019 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:41.321409 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:41.321414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:41.321436 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:41.321730 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.321738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.322742 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.322749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.324163 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.324175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.325366 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.325372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.325546 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.325571 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:58:41.325682 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 20:58:41.325824 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1779,"mutex_wait_us":2,"thread_start_us":1567,"threads_started":1}
I20260923 20:58:41.327260 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511a17d308ff40eda3d4d6a9c817ff5d" candidate_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0015fe049a2451abc4839084a1bac49" is_pre_election: true
I20260923 20:58:41.327792 20446 raft_consensus.cc:2454] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b771d5b64e4245f8bb1c090f1c4658c0 in term 0.
I20260923 20:58:41.328568 20513 leader_election.cc:302] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b771d5b64e4245f8bb1c090f1c4658c0, d0015fe049a2451abc4839084a1bac49; no voters:
I20260923 20:58:41.329027 20778 raft_consensus.cc:2781] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:58:41.329180 20778 raft_consensus.cc:482] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:58:41.329311 20778 raft_consensus.cc:3037] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:41.329466 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.318843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46297 (local address 127.0.97.2:40513)
0923 20:58:41.319743 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.319776 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 20:58:41.319968 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:58:41.320684 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:58:41.320688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:58:41.320728 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:58:41.321125 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.321140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.323301 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.323309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.325966 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:58:41.325984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.328501 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.328508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:58:41.328717 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.328756 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:58:41.328903 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:58:41.329065 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":1236,"threads_started":1}
I20260923 20:58:41.329476 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.319715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40513 (local address 127.0.97.3:46297)
0923 20:58:41.319939 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.319947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.319970 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:58:41.320199 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.320210 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.320335 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.320427 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:58:41.321303 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.322675 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.326712 (+ 4037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.328289 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.328468 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.328615 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.328949 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.329140 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:58:41.330806 20707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511a17d308ff40eda3d4d6a9c817ff5d" candidate_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473400b17f614d9cb54461351ea92885" is_pre_election: true
I20260923 20:58:41.331395 20707 raft_consensus.cc:2454] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b771d5b64e4245f8bb1c090f1c4658c0 in term 0.
I20260923 20:58:41.332444 20778 raft_consensus.cc:504] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.333112 20778 leader_election.cc:288] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [CANDIDATE]: Term 1 election: Requested vote from peers 473400b17f614d9cb54461351ea92885 (127.0.97.3:46297), d0015fe049a2451abc4839084a1bac49 (127.0.97.1:40331)
I20260923 20:58:41.333668 20707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511a17d308ff40eda3d4d6a9c817ff5d" candidate_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473400b17f614d9cb54461351ea92885"
I20260923 20:58:41.333765 20446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511a17d308ff40eda3d4d6a9c817ff5d" candidate_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0015fe049a2451abc4839084a1bac49"
I20260923 20:58:41.333966 20707 raft_consensus.cc:3037] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:41.334051 20446 raft_consensus.cc:3037] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:41.336587 20446 raft_consensus.cc:2454] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b771d5b64e4245f8bb1c090f1c4658c0 in term 1.
I20260923 20:58:41.337111 20513 leader_election.cc:302] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b771d5b64e4245f8bb1c090f1c4658c0, d0015fe049a2451abc4839084a1bac49; no voters:
I20260923 20:58:41.337445 20778 raft_consensus.cc:2781] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:41.338012 20778 raft_consensus.cc:686] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 1 LEADER]: Becoming Leader. State: Replica: b771d5b64e4245f8bb1c090f1c4658c0, State: Running, Role: LEADER
I20260923 20:58:41.337998 20707 raft_consensus.cc:2454] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b771d5b64e4245f8bb1c090f1c4658c0 in term 1.
I20260923 20:58:41.338960 20778 consensus_queue.cc:237] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } }
I20260923 20:58:41.344588 20303 catalog_manager.cc:5495] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b771d5b64e4245f8bb1c090f1c4658c0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b771d5b64e4245f8bb1c090f1c4658c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 } health_report { overall_health: UNKNOWN } } }
I20260923 20:58:41.391081 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.385336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.0.97.2:38475)
0923 20:58:41.385532 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.385540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.385568 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:58:41.385884 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.385890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.386005 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.386167 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:58:41.386542 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.387259 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.388810 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.389231 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.389370 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.389960 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:58:41.390490 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:58:41.390654 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.390714 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.390801 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:58:41.400118 20446 raft_consensus.cc:1261] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 1 FOLLOWER]: Refusing update from remote peer b771d5b64e4245f8bb1c090f1c4658c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:41.400118 20707 raft_consensus.cc:1261] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 1 FOLLOWER]: Refusing update from remote peer b771d5b64e4245f8bb1c090f1c4658c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:58:41.401284 20778 consensus_queue.cc:1035] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0015fe049a2451abc4839084a1bac49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:41.401604 20781 consensus_queue.cc:1035] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "473400b17f614d9cb54461351ea92885" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:58:41.420395 20785 mvcc.cc:205] Tried to move back new op lower bound from 7332647409242464256 to 7332647409008902144. Current Snapshot: MvccSnapshot[applied={T|T < 7332647409242464256}]
I20260923 20:58:41.447224 20303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:41.447552 20303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:58:41.451272 20303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8057c3f0f4a54c06a8cdb6ed124f9adb: Sending DeleteTablet for 3 replicas of tablet 511a17d308ff40eda3d4d6a9c817ff5d
I20260923 20:58:41.452742 20687 tablet_service.cc:1514] Processing DeleteTablet for tablet 511a17d308ff40eda3d4d6a9c817ff5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:41 UTC) from {username='slave'} at 127.0.0.1:46798
I20260923 20:58:41.452739 20426 tablet_service.cc:1514] Processing DeleteTablet for tablet 511a17d308ff40eda3d4d6a9c817ff5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:41 UTC) from {username='slave'} at 127.0.0.1:40024
I20260923 20:58:41.453539 20557 tablet_service.cc:1514] Processing DeleteTablet for tablet 511a17d308ff40eda3d4d6a9c817ff5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:58:41 UTC) from {username='slave'} at 127.0.0.1:40544
I20260923 20:58:41.453650 20794 tablet_replica.cc:330] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: stopping tablet replica
I20260923 20:58:41.454216 20794 raft_consensus.cc:2229] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:41.455011 20794 raft_consensus.cc:2258] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:41.456010 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20364
I20260923 20:58:41.457423 20794 ts_tablet_manager.cc:1905] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:41.466939 20794 ts_tablet_manager.cc:1918] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:41.467281 20794 log.cc:1196] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-2/wal/wals/511a17d308ff40eda3d4d6a9c817ff5d
I20260923 20:58:41.467952 20794 ts_tablet_manager.cc:1939] T 511a17d308ff40eda3d4d6a9c817ff5d P 473400b17f614d9cb54461351ea92885: Deleting consensus metadata
I20260923 20:58:41.470374 20291 catalog_manager.cc:4841] TS 473400b17f614d9cb54461351ea92885 (127.0.97.3:46297): tablet 511a17d308ff40eda3d4d6a9c817ff5d (table test-table [id=09541cfc41724e1ab79b25838d7ea201]) successfully deleted
I20260923 20:58:41.484884 20796 tablet_replica.cc:330] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: stopping tablet replica
I20260923 20:58:41.485462 20796 raft_consensus.cc:2229] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:58:41.486496 20796 raft_consensus.cc:2258] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:41.494218 20796 ts_tablet_manager.cc:1905] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:41.497710 20795 tablet_replica.cc:330] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: stopping tablet replica
I20260923 20:58:41.498292 20795 raft_consensus.cc:2229] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:58:41.498926 20795 raft_consensus.cc:2258] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:58:41.501217 20795 ts_tablet_manager.cc:1905] T 511a17d308ff40eda3d4d6a9c817ff5d P d0015fe049a2451abc4839084a1bac49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:58:41.503679 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.457201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.0.97.1:40331)
0923 20:58:41.457381 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.457387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.481716 (+ 24329us) server_negotiation.cc:407] Connection header received
0923 20:58:41.483710 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.483720 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.483864 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.483962 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:41.486962 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.488594 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.497519 (+ 8925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.498253 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.498385 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.501837 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:41.501852 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:41.501872 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:41.502664 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:41.502731 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:41.502758 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:41.502873 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.503132 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.503266 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:58:41.505154 20486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:41.508677 20796 ts_tablet_manager.cc:1918] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:58:41.508885 20796 log.cc:1196] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Deleting WAL directory at /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790196790989944-388-0/minicluster-data/ts-1/wal/wals/511a17d308ff40eda3d4d6a9c817ff5d
I20260923 20:58:41.509421 20796 ts_tablet_manager.cc:1939] T 511a17d308ff40eda3d4d6a9c817ff5d P b771d5b64e4245f8bb1c090f1c4658c0: Deleting consensus metadata
I20260923 20:58:41.511121 20289 catalog_manager.cc:4841] TS b771d5b64e4245f8bb1c090f1c4658c0 (127.0.97.2:38475): tablet 511a17d308ff40eda3d4d6a9c817ff5d (table test-table [id=09541cfc41724e1ab79b25838d7ea201]) successfully deleted
I20260923 20:58:41.606396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 20364
I20260923 20:58:41.624883 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20495
W20260923 20:58:41.624928 20292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40331 (error 108)
I20260923 20:58:41.627691 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.626518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40048)
0923 20:58:41.627340 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.627485 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":562,"threads_started":1}
W20260923 20:58:41.627991 20292 catalog_manager.cc:4584] TS d0015fe049a2451abc4839084a1bac49 (127.0.97.1:40331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 511a17d308ff40eda3d4d6a9c817ff5d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
I20260923 20:58:41.633039 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.626013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.0.97.2:38475)
0923 20:58:41.626226 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.626232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.626253 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:58:41.626318 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.626323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.626456 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.626587 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:41.626982 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.628217 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.629522 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.630288 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.630463 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.631595 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:41.631604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:41.631618 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:41.632242 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:41.632293 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:41.632319 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:41.632453 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.632581 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.632693 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:58:41.633728 20617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:41.681734 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.681265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40052)
0923 20:58:41.681511 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.681596 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 20:58:41.683068 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.682551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40060)
0923 20:58:41.682765 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.682891 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:58:41.743614 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.743058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40074)
0923 20:58:41.743316 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.743429 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260923 20:58:41.744849 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.744372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40084)
0923 20:58:41.744557 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.744645 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 20:58:41.751475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 20495
I20260923 20:58:41.771128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20625
I20260923 20:58:41.778352 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.771928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.0.97.3:46297)
0923 20:58:41.772132 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:41.772136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:41.772156 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:58:41.772314 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:41.772318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:41.772410 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:41.772503 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:41.773093 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.773922 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.775262 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:41.775750 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:41.775856 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:41.776905 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:41.776912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:41.776926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:41.777510 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:41.777557 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:41.777576 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:41.777665 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:41.777794 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 20:58:41.777966 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:58:41.778996 20747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:41.825670 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.825075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40090)
0923 20:58:41.825365 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.825487 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 20:58:41.826947 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:41.826461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40096)
0923 20:58:41.826693 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:41.826783 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260923 20:58:41.871186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 20625
I20260923 20:58:41.890738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20274
I20260923 20:58:41.891881 20335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:42.003762 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:42.003245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40331 (local address 127.0.0.1:40104)
0923 20:58:42.003451 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 20:58:42.003565 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 20:58:42.003805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 20274
2026-09-23T20:58:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3144 ms)
[----------] 6 tests from EncryptionPolicyTest (22655 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:42 dist-test-slave-plcg krb5kdc[20806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:42 dist-test-slave-plcg krb5kdc[20806](info): set up 2 sockets
Sep 23 20:58:42 dist-test-slave-plcg krb5kdc[20806](info): commencing operation
krb5kdc: starting...
W20260923 20:58:43.126286 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:43 dist-test-slave-plcg krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:43.394728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:43.734709 20819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:43.735029 20819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:43.735199 20819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:43.742823 20819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:43.742954 20819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:43.743058 20819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:43.743171 20819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:43.743237 20819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:43.753396 20819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:43.754846 20819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:43.756250 20819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:43.768795 20825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:43.768818 20827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:43.768620 20824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:43.769932 20819 server_base.cc:1034] running on GCE node
I20260923 20:58:43.770771 20819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:43.772260 20819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:43.773481 20819 hybrid_clock.cc:648] HybridClock initialized: now 1790197123773437 us; error 35 us; skew 500 ppm
I20260923 20:58:43.773872 20819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:43 dist-test-slave-plcg krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:43.777094 20819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:43.778784 20819 webserver.cc:460] Webserver started at http://127.0.0.1:32847/ using document root <none> and password file <none>
I20260923 20:58:43.779570 20819 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:43.779717 20819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:43.780068 20819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:43.783340 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "36059b5ebb72427aa9d0f3c5d00092ef"
format_stamp: "Formatted at 2026-09-23 20:58:43 on dist-test-slave-plcg"
server_key: "fc75069504daea5fbacf51da01f1c548"
server_key_iv: "3ed046f7d431ac7de8cb6b76e37b60a1"
server_key_version: "encryptionkey@0"
I20260923 20:58:43.784215 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "36059b5ebb72427aa9d0f3c5d00092ef"
format_stamp: "Formatted at 2026-09-23 20:58:43 on dist-test-slave-plcg"
server_key: "fc75069504daea5fbacf51da01f1c548"
server_key_iv: "3ed046f7d431ac7de8cb6b76e37b60a1"
server_key_version: "encryptionkey@0"
I20260923 20:58:43.790167 20819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:58:43.794090 20834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:43.795598 20819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:43.795828 20819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "36059b5ebb72427aa9d0f3c5d00092ef"
format_stamp: "Formatted at 2026-09-23 20:58:43 on dist-test-slave-plcg"
server_key: "fc75069504daea5fbacf51da01f1c548"
server_key_iv: "3ed046f7d431ac7de8cb6b76e37b60a1"
server_key_version: "encryptionkey@0"
I20260923 20:58:43.796072 20819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:43.810891 20819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:43.815397 20819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:43.815724 20819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:43.835431 20819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44057
I20260923 20:58:43.835453 20885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44057 every 8 connection(s)
I20260923 20:58:43.836951 20819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:43.840889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 20819
I20260923 20:58:43.840969 20886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:43.841146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:43.841930 388 external_mini_cluster.cc:1428] Setting key d65f2cbf2ef0c07590e57bf02bdbef62
I20260923 20:58:43.855787 20886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef: Bootstrap starting.
Sep 23 20:58:43 dist-test-slave-plcg krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:58:43.859723 20886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:43.860873 20886 log.cc:824] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:43.864145 20886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef: No bootstrap required, opened a new log
I20260923 20:58:43.867722 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:43.843855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36159 (local address 127.0.0.1:44057)
0923 20:58:43.844728 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:43.844740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:43.844782 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:58:43.847276 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:43.847307 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:43.847795 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:43.848177 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:43.850034 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:43.851387 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:43.853367 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:43.854188 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:43.854327 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:43.859356 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:43.859366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:43.859383 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:43.859431 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:43.863160 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:43.864066 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:43.864073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:43.864081 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:43.864250 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:43.864772 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:43.864776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:43.864778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:43.865333 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:43.865677 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:43.866296 (+ 619us) server_negotiation.cc:299] Negotiation successful
0923 20:58:43.866648 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":390,"threads_started":1}
I20260923 20:58:43.871969 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:43.872840 20886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44057 } }
I20260923 20:58:43.873250 20886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:43.873353 20886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36059b5ebb72427aa9d0f3c5d00092ef, State: Initialized, Role: FOLLOWER
I20260923 20:58:43.874047 20886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44057 } }
I20260923 20:58:43.874302 20886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:43.874452 20886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:43.874634 20886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:43.877413 20886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44057 } }
I20260923 20:58:43.878139 20886 leader_election.cc:302] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36059b5ebb72427aa9d0f3c5d00092ef; no voters:
I20260923 20:58:43.878804 20886 leader_election.cc:288] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:43.878911 20896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:43.881592 20886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:43.882097 20896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [term 1 LEADER]: Becoming Leader. State: Replica: 36059b5ebb72427aa9d0f3c5d00092ef, State: Running, Role: LEADER
I20260923 20:58:43.882781 20896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44057 } }
I20260923 20:58:43.895535 20897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44057 } } }
I20260923 20:58:43.895993 20897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:43.896745 20898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36059b5ebb72427aa9d0f3c5d00092ef. Latest consensus state: current_term: 1 leader_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36059b5ebb72427aa9d0f3c5d00092ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44057 } } }
I20260923 20:58:43.897094 20898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:43.899623 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:43.876995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60125 (local address 127.0.0.1:44057)
0923 20:58:43.877201 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:43.877209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:43.877651 (+ 442us) server_negotiation.cc:407] Connection header received
0923 20:58:43.877894 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:43.877901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:43.877975 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:43.878118 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:43.879868 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:43.880751 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:43.882439 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:43.883194 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:43.883315 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:43.887236 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:43.887246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:43.887253 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:43.887308 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:43.894734 (+ 7426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:43.896092 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:43.896097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:43.896100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:43.896278 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:43.896960 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:43.896970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:43.896975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:43.897519 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:43.897892 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:43.898797 (+ 905us) server_negotiation.cc:299] Negotiation successful
0923 20:58:43.899064 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:58:43.904526 20904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:43.910452 20904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:43.920620 20904 catalog_manager.cc:1347] Generated new cluster ID: 274be854f1b84e06bf222e8d49632036
I20260923 20:58:43.920758 20904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:43.945109 20904 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:43.946676 20904 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:43.971921 20904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36059b5ebb72427aa9d0f3c5d00092ef: Generated new TSK 0
I20260923 20:58:43.972904 20904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:43.992715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20819
I20260923 20:58:43.994738 20881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:44.091279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 20819
W20260923 20:58:44.109443 20804 connection.cc:535] client connection to 127.0.0.1:44057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44057): Connection reset by peer (error 104)
2026-09-23T20:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:44 dist-test-slave-plcg krb5kdc[20918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:44 dist-test-slave-plcg krb5kdc[20918](info): set up 2 sockets
Sep 23 20:58:44 dist-test-slave-plcg krb5kdc[20918](info): commencing operation
krb5kdc: starting...
W20260923 20:58:45.194408 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:45 dist-test-slave-plcg krb5kdc[20918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:45.470494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:45.818415 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:45.818737 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:45.818912 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:45.827353 20931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:45.827504 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:45.827646 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:45.827736 20931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:45.827783 20931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:45.839762 20931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:45.841140 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:45.842828 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:45.857226 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:45.857302 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:45.857180 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:45.864058 20931 server_base.cc:1034] running on GCE node
I20260923 20:58:45.865242 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:45.867046 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:45.868352 20931 hybrid_clock.cc:648] HybridClock initialized: now 1790197125868248 us; error 67 us; skew 500 ppm
I20260923 20:58:45.869027 20931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:45 dist-test-slave-plcg krb5kdc[20918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:45.876747 20931 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:45.878679 20931 webserver.cc:460] Webserver started at http://127.0.0.1:45513/ using document root <none> and password file <none>
I20260923 20:58:45.879478 20931 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:45.879633 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:45.879982 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:45.883371 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "628b0e58ed3247d88c213d3bf7f2a8fa"
format_stamp: "Formatted at 2026-09-23 20:58:45 on dist-test-slave-plcg"
server_key: "155cb68fb137c6402034aafeac2cca16"
server_key_iv: "a4f6103bb29260fffd0f17d605923c23"
server_key_version: "encryptionkey@0"
I20260923 20:58:45.884223 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "628b0e58ed3247d88c213d3bf7f2a8fa"
format_stamp: "Formatted at 2026-09-23 20:58:45 on dist-test-slave-plcg"
server_key: "155cb68fb137c6402034aafeac2cca16"
server_key_iv: "a4f6103bb29260fffd0f17d605923c23"
server_key_version: "encryptionkey@0"
I20260923 20:58:45.890872 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:58:45.895870 20946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:45.898212 20931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 20:58:45.898461 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "628b0e58ed3247d88c213d3bf7f2a8fa"
format_stamp: "Formatted at 2026-09-23 20:58:45 on dist-test-slave-plcg"
server_key: "155cb68fb137c6402034aafeac2cca16"
server_key_iv: "a4f6103bb29260fffd0f17d605923c23"
server_key_version: "encryptionkey@0"
I20260923 20:58:45.898712 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:45.918675 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:45.923379 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:45.923715 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:45.946195 20931 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36573
I20260923 20:58:45.946229 20997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36573 every 8 connection(s)
I20260923 20:58:45.947845 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:45.951984 20998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:45.957769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 20931
I20260923 20:58:45.957993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:45.958535 388 external_mini_cluster.cc:1428] Setting key 3f769ca59b1dec6a0a1e80d48606e03c
I20260923 20:58:45.964511 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa: Bootstrap starting.
I20260923 20:58:45.968544 20998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:45.969894 20998 log.cc:824] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:45.973112 20998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa: No bootstrap required, opened a new log
I20260923 20:58:45.974299 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:45.961152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42493 (local address 127.0.0.1:36573)
0923 20:58:45.962249 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:45.962265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:45.962310 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:58:45.964148 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:45.964174 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:45.964667 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:45.965062 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:45.965681 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:45.967122 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:45.969214 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:45.970091 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:45.970267 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:45.971729 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:45.971737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:45.971754 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:45.972356 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:45.972441 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:45.972471 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:45.972666 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:45.972965 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 20:58:45.973153 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":525,"threads_started":1}
I20260923 20:58:45.978902 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:45.982318 20998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260923 20:58:45.982781 20998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:45.982901 20998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628b0e58ed3247d88c213d3bf7f2a8fa, State: Initialized, Role: FOLLOWER
I20260923 20:58:45.983669 20998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260923 20:58:45.983971 20998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:45.984143 20998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:45.984365 20998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:45.987547 20998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260923 20:58:45.988240 20998 leader_election.cc:302] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628b0e58ed3247d88c213d3bf7f2a8fa; no voters:
I20260923 20:58:45.988972 20998 leader_election.cc:288] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:45.989166 21008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:45.990127 21008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [term 1 LEADER]: Becoming Leader. State: Replica: 628b0e58ed3247d88c213d3bf7f2a8fa, State: Running, Role: LEADER
I20260923 20:58:45.990855 21008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } }
I20260923 20:58:45.991797 20998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:45.991879 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:45.983545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47325 (local address 127.0.0.1:36573)
0923 20:58:45.983756 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:45.983762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:45.984114 (+ 352us) server_negotiation.cc:407] Connection header received
0923 20:58:45.984347 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:45.984356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:45.984427 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:45.984542 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:45.985212 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:45.986213 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:45.988198 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:45.988835 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:45.989023 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:45.990510 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:45.990516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:45.990521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:45.990543 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:45.990616 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:45.990637 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:45.990761 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:45.991485 (+ 724us) server_negotiation.cc:299] Negotiation successful
0923 20:58:45.991545 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:58:45.995256 21010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } } }
I20260923 20:58:45.995787 21010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:45.996652 21009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628b0e58ed3247d88c213d3bf7f2a8fa. Latest consensus state: current_term: 1 leader_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628b0e58ed3247d88c213d3bf7f2a8fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36573 } } }
I20260923 20:58:45.997072 21009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:45.998261 21016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:46.004292 21016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:46.014652 21016 catalog_manager.cc:1347] Generated new cluster ID: 269498253158452e847c12bf77f3e01c
I20260923 20:58:46.014806 21016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:46.042443 21016 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:46.043843 21016 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:46.051832 21016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 628b0e58ed3247d88c213d3bf7f2a8fa: Generated new TSK 0
I20260923 20:58:46.052996 21016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:46.064811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20931
W20260923 20:58:46.066154 20993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42493
W20260923 20:58:46.067015 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 20931: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:58:46.067235 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 20931
W20260923 20:58:46.082211 20916 connection.cc:535] client connection to 127.0.0.1:36573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36573): Connection reset by peer (error 104)
2026-09-23T20:58:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:46 dist-test-slave-plcg krb5kdc[21029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:46 dist-test-slave-plcg krb5kdc[21029](info): set up 2 sockets
Sep 23 20:58:46 dist-test-slave-plcg krb5kdc[21029](info): commencing operation
krb5kdc: starting...
W20260923 20:58:47.170195 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:47 dist-test-slave-plcg krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:47.435149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:47.779240 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:47.779564 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:47.779700 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:47.787129 21042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:47.787257 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:47.787354 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:47.787467 21042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:47.787555 21042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:47.803417 21042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:47.805238 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:47.807080 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:47.822294 21048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:47.822279 21047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:47.822904 21050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:47.823302 21042 server_base.cc:1034] running on GCE node
I20260923 20:58:47.824534 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:47.826689 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:47.827926 21042 hybrid_clock.cc:648] HybridClock initialized: now 1790197127827865 us; error 59 us; skew 500 ppm
I20260923 20:58:47.828478 21042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:47 dist-test-slave-plcg krb5kdc[21029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:47.832567 21042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:47.834592 21042 webserver.cc:460] Webserver started at http://127.0.0.1:41497/ using document root <none> and password file <none>
I20260923 20:58:47.835354 21042 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:47.835523 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:47.835923 21042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:47.840623 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "273593d1d5d34b76b587053e4a599c74"
format_stamp: "Formatted at 2026-09-23 20:58:47 on dist-test-slave-plcg"
server_key: "d1e5c87c297319baababd6eabfb6d727"
server_key_iv: "55b9a8187c80019475b8bb8c9b0de386"
server_key_version: "encryptionkey@0"
I20260923 20:58:47.841508 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "273593d1d5d34b76b587053e4a599c74"
format_stamp: "Formatted at 2026-09-23 20:58:47 on dist-test-slave-plcg"
server_key: "d1e5c87c297319baababd6eabfb6d727"
server_key_iv: "55b9a8187c80019475b8bb8c9b0de386"
server_key_version: "encryptionkey@0"
I20260923 20:58:47.847963 21042 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 20:58:47.852101 21057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:47.853873 21042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 20:58:47.854105 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "273593d1d5d34b76b587053e4a599c74"
format_stamp: "Formatted at 2026-09-23 20:58:47 on dist-test-slave-plcg"
server_key: "d1e5c87c297319baababd6eabfb6d727"
server_key_iv: "55b9a8187c80019475b8bb8c9b0de386"
server_key_version: "encryptionkey@0"
I20260923 20:58:47.854341 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:47.860941 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:47.865525 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:47.865931 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:47.886284 21042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36619
I20260923 20:58:47.886397 21108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36619 every 8 connection(s)
I20260923 20:58:47.887863 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:47.891454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21042
I20260923 20:58:47.891645 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:47.892053 388 external_mini_cluster.cc:1428] Setting key fbcfe256035933908181fcc0959cfd0d
I20260923 20:58:47.894101 21109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:47.901075 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:47.894029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47033 (local address 127.0.0.1:36619)
0923 20:58:47.895341 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:47.895353 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:47.895395 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:58:47.897114 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:47.897139 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:47.897535 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:47.898043 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:47.899150 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:47.899156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:47.899188 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:47.899207 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:47.899281 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:47.899300 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:47.899690 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:47.899888 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 20:58:47.900099 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028,"thread_start_us":754,"threads_started":1}
I20260923 20:58:47.905361 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:47.907303 21109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74: Bootstrap starting.
I20260923 20:58:47.911218 21109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:47.912494 21109 log.cc:824] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:47.912665 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:47.910047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45175 (local address 127.0.0.1:36619)
0923 20:58:47.910282 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:47.910288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:47.910562 (+ 274us) server_negotiation.cc:407] Connection header received
0923 20:58:47.910781 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:47.910785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:47.910834 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:47.910937 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:47.911922 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:47.911928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:47.911932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:47.911951 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:47.912003 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:47.912025 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:47.912145 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:47.912277 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 20:58:47.912348 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 20:58:47.915620 21109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74: No bootstrap required, opened a new log
I20260923 20:58:47.923244 21109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273593d1d5d34b76b587053e4a599c74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36619 } }
I20260923 20:58:47.923732 21109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:47.923844 21109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273593d1d5d34b76b587053e4a599c74, State: Initialized, Role: FOLLOWER
I20260923 20:58:47.924515 21109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273593d1d5d34b76b587053e4a599c74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36619 } }
I20260923 20:58:47.924779 21109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:47.924930 21109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:47.925124 21109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:47.927464 21109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273593d1d5d34b76b587053e4a599c74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36619 } }
I20260923 20:58:47.928037 21109 leader_election.cc:302] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273593d1d5d34b76b587053e4a599c74; no voters:
I20260923 20:58:47.928670 21109 leader_election.cc:288] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:47.928856 21119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:47.930007 21119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [term 1 LEADER]: Becoming Leader. State: Replica: 273593d1d5d34b76b587053e4a599c74, State: Running, Role: LEADER
I20260923 20:58:47.930790 21119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273593d1d5d34b76b587053e4a599c74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36619 } }
I20260923 20:58:47.931106 21109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:47.933648 21120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273593d1d5d34b76b587053e4a599c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273593d1d5d34b76b587053e4a599c74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36619 } } }
I20260923 20:58:47.933686 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273593d1d5d34b76b587053e4a599c74. Latest consensus state: current_term: 1 leader_uuid: "273593d1d5d34b76b587053e4a599c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273593d1d5d34b76b587053e4a599c74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36619 } } }
I20260923 20:58:47.934259 21120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:47.934259 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:47.935526 21128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:47.941887 21128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:47.952077 21128 catalog_manager.cc:1347] Generated new cluster ID: 1a828bb57f07457a97ad0aec975bf387
I20260923 20:58:47.952229 21128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:47.988457 21128 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:47.988952 21128 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:47.995918 21128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273593d1d5d34b76b587053e4a599c74: Generated new TSK 0
I20260923 20:58:47.996819 21128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:48.012333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21042
W20260923 20:58:48.013562 21104 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47033
W20260923 20:58:48.014480 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21042: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:58:48.014639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21042
2026-09-23T20:58:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:48 dist-test-slave-plcg krb5kdc[21140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:48 dist-test-slave-plcg krb5kdc[21140](info): set up 2 sockets
Sep 23 20:58:48 dist-test-slave-plcg krb5kdc[21140](info): commencing operation
krb5kdc: starting...
W20260923 20:58:49.130386 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:49 dist-test-slave-plcg krb5kdc[21140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:49.402380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:49.746311 21153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:49.746635 21153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:49.746822 21153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:49.754532 21153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:49.754660 21153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:49.754773 21153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:49.754884 21153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:49.754966 21153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:49.765298 21153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:49.766714 21153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:49.768060 21153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:49.780817 21159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:49.780835 21161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:49.780886 21158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:49.781764 21153 server_base.cc:1034] running on GCE node
I20260923 20:58:49.782684 21153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:49.784149 21153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:49.785395 21153 hybrid_clock.cc:648] HybridClock initialized: now 1790197129785357 us; error 39 us; skew 500 ppm
I20260923 20:58:49.785818 21153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:49 dist-test-slave-plcg krb5kdc[21140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:49.789248 21153 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:49.791003 21153 webserver.cc:460] Webserver started at http://127.0.0.1:39413/ using document root <none> and password file <none>
I20260923 20:58:49.791778 21153 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:49.791940 21153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:49.792270 21153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:49.795564 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5"
format_stamp: "Formatted at 2026-09-23 20:58:49 on dist-test-slave-plcg"
server_key: "320d4074e1d6300a79a792867524c67b"
server_key_iv: "7770941d437796a5446f0e565661fe4a"
server_key_version: "encryptionkey@0"
I20260923 20:58:49.796450 21153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5"
format_stamp: "Formatted at 2026-09-23 20:58:49 on dist-test-slave-plcg"
server_key: "320d4074e1d6300a79a792867524c67b"
server_key_iv: "7770941d437796a5446f0e565661fe4a"
server_key_version: "encryptionkey@0"
I20260923 20:58:49.802259 21153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:58:49.806303 21168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:49.807821 21153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:49.808043 21153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5"
format_stamp: "Formatted at 2026-09-23 20:58:49 on dist-test-slave-plcg"
server_key: "320d4074e1d6300a79a792867524c67b"
server_key_iv: "7770941d437796a5446f0e565661fe4a"
server_key_version: "encryptionkey@0"
I20260923 20:58:49.808277 21153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:49.821908 21153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:49.826668 21153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:49.826989 21153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:49.846504 21153 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40401
I20260923 20:58:49.846525 21219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40401 every 8 connection(s)
I20260923 20:58:49.848120 21153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:49.852144 21220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:49.858505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21153
I20260923 20:58:49.858717 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:49.859160 388 external_mini_cluster.cc:1428] Setting key 18276a5ecbfc1a20538db8ac5f0eec51
I20260923 20:58:49.864040 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5: Bootstrap starting.
I20260923 20:58:49.867640 21220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:49.868762 21220 log.cc:824] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:49.871863 21220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5: No bootstrap required, opened a new log
Sep 23 20:58:49 dist-test-slave-plcg krb5kdc[21140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:58:49.880158 21220 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260923 20:58:49.880578 21220 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:49.880699 21220 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca53f4a2a03b4ea6b1208d1ee7dfd7d5, State: Initialized, Role: FOLLOWER
I20260923 20:58:49.881412 21220 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260923 20:58:49.881704 21220 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:49.881906 21220 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:49.882109 21220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:49.883267 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:49.861051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42215 (local address 127.0.0.1:40401)
0923 20:58:49.862006 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:49.862019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:49.862068 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:58:49.863845 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:49.863869 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:49.864377 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:49.864804 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:49.866392 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:49.867778 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:49.869229 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:49.870082 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:49.870309 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:49.874829 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:49.874840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:49.874857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:49.874903 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:49.878399 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:49.879198 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:49.879209 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:49.879217 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:49.879412 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:49.879840 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:49.879845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:49.879850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:49.880504 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:49.880851 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:49.881298 (+ 447us) server_negotiation.cc:299] Negotiation successful
0923 20:58:49.881812 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":408,"threads_started":1}
I20260923 20:58:49.885830 21220 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260923 20:58:49.886571 21220 leader_election.cc:302] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca53f4a2a03b4ea6b1208d1ee7dfd7d5; no voters:
I20260923 20:58:49.888053 21220 leader_election.cc:288] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:49.888219 21225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:49.889372 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:49.892357 21220 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:49.894186 21225 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [term 1 LEADER]: Becoming Leader. State: Replica: ca53f4a2a03b4ea6b1208d1ee7dfd7d5, State: Running, Role: LEADER
I20260923 20:58:49.895083 21225 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } }
I20260923 20:58:49.898784 21227 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca53f4a2a03b4ea6b1208d1ee7dfd7d5. Latest consensus state: current_term: 1 leader_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } } }
I20260923 20:58:49.899331 21227 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:49.900005 21226 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca53f4a2a03b4ea6b1208d1ee7dfd7d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40401 } } }
I20260923 20:58:49.900385 21226 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:49.900671 21236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:49.915988 21236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:49.931267 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:49.907722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51835 (local address 127.0.0.1:40401)
0923 20:58:49.907924 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:49.907933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:49.911030 (+ 3097us) server_negotiation.cc:407] Connection header received
0923 20:58:49.911262 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:49.911267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:49.911353 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:49.911470 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:49.913508 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:49.914703 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:49.916304 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:49.916799 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:49.916958 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:49.920731 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:49.920742 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:49.920750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:49.920813 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:49.925907 (+ 5094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:49.927224 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:49.927234 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:49.927244 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:49.927459 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:49.928041 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:49.928050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:49.928055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:49.928701 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:49.929055 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:49.930671 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0923 20:58:49.930986 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:58:49.932821 21236 catalog_manager.cc:1347] Generated new cluster ID: ff3af40985a64876904eec7571cccaa5
I20260923 20:58:49.932947 21236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:49.944391 21236 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:49.945937 21236 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:49.960074 21236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca53f4a2a03b4ea6b1208d1ee7dfd7d5: Generated new TSK 0
I20260923 20:58:49.961325 21236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:49.973412 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21153
I20260923 20:58:49.975117 21215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:50.068662 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21153
2026-09-23T20:58:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2054 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:50 dist-test-slave-plcg krb5kdc[21252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:50 dist-test-slave-plcg krb5kdc[21252](info): set up 2 sockets
Sep 23 20:58:50 dist-test-slave-plcg krb5kdc[21252](info): commencing operation
krb5kdc: starting...
W20260923 20:58:51.174458 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:51 dist-test-slave-plcg krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:51.441962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:51.782781 21265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:51.783110 21265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:51.783306 21265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:51.790900 21265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:51.791085 21265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:51.791177 21265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:51.791308 21265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:51.791390 21265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:51.801839 21265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:51.803256 21265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:51.804693 21265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:51.817148 21270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:51.817189 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:51.817516 21273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:51.818602 21265 server_base.cc:1034] running on GCE node
I20260923 20:58:51.819458 21265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:51.821038 21265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:51.822286 21265 hybrid_clock.cc:648] HybridClock initialized: now 1790197131822255 us; error 64 us; skew 500 ppm
I20260923 20:58:51.822679 21265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:51 dist-test-slave-plcg krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:51.829334 21265 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:51.831079 21265 webserver.cc:460] Webserver started at http://127.0.0.1:37785/ using document root <none> and password file <none>
I20260923 20:58:51.831841 21265 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:51.832012 21265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:51.832340 21265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:51.835580 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "f602f9bc505c49f7b0a71a1cd05d3e13"
format_stamp: "Formatted at 2026-09-23 20:58:51 on dist-test-slave-plcg"
server_key: "96c6d824ed1b93816d1ff317b4d603e2"
server_key_iv: "6fe9b6324a4c08866d7b93d4c8b1b286"
server_key_version: "encryptionkey@0"
I20260923 20:58:51.836443 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "f602f9bc505c49f7b0a71a1cd05d3e13"
format_stamp: "Formatted at 2026-09-23 20:58:51 on dist-test-slave-plcg"
server_key: "96c6d824ed1b93816d1ff317b4d603e2"
server_key_iv: "6fe9b6324a4c08866d7b93d4c8b1b286"
server_key_version: "encryptionkey@0"
I20260923 20:58:51.842226 21265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:58:51.845965 21280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:51.847395 21265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:51.847621 21265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "f602f9bc505c49f7b0a71a1cd05d3e13"
format_stamp: "Formatted at 2026-09-23 20:58:51 on dist-test-slave-plcg"
server_key: "96c6d824ed1b93816d1ff317b4d603e2"
server_key_iv: "6fe9b6324a4c08866d7b93d4c8b1b286"
server_key_version: "encryptionkey@0"
I20260923 20:58:51.847853 21265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:51.877592 21265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:51.882519 21265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:51.882965 21265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:51.908452 21265 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46387
I20260923 20:58:51.908468 21331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46387 every 8 connection(s)
I20260923 20:58:51.910033 21265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:51.913998 21332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:51.918531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21265
I20260923 20:58:51.918725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:51.919133 388 external_mini_cluster.cc:1428] Setting key bcecf20ec731b9ab4735d93d9efc29c8
I20260923 20:58:51.927071 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13: Bootstrap starting.
I20260923 20:58:51.931063 21332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:51.932463 21332 log.cc:824] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:51.935822 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:51.921023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46425 (local address 127.0.0.1:46387)
0923 20:58:51.921895 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:51.921914 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:51.921968 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:58:51.925084 (+ 3116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:51.925119 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:51.925699 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:51.926223 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:51.926771 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:51.928744 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:51.930316 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:51.931438 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:51.931666 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:51.932857 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:51.932869 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:51.932897 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:51.933647 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:51.933738 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:51.933765 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:51.933988 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:51.934217 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:58:51.934508 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":353,"threads_started":1}
I20260923 20:58:51.936146 21332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13: No bootstrap required, opened a new log
I20260923 20:58:51.940002 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:51.945129 21332 raft_consensus.cc:348] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46387 } }
I20260923 20:58:51.945681 21332 raft_consensus.cc:374] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:51.945806 21332 raft_consensus.cc:729] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f602f9bc505c49f7b0a71a1cd05d3e13, State: Initialized, Role: FOLLOWER
I20260923 20:58:51.946507 21332 consensus_queue.cc:260] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46387 } }
I20260923 20:58:51.946792 21332 raft_consensus.cc:388] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:51.946949 21332 raft_consensus.cc:482] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:51.947141 21332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:51.949470 21332 raft_consensus.cc:504] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46387 } }
I20260923 20:58:51.950209 21332 leader_election.cc:302] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f602f9bc505c49f7b0a71a1cd05d3e13; no voters:
I20260923 20:58:51.950939 21332 leader_election.cc:288] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:51.951066 21342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:51.951433 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:51.944517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41783 (local address 127.0.0.1:46387)
0923 20:58:51.944747 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:51.944753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:51.945076 (+ 323us) server_negotiation.cc:407] Connection header received
0923 20:58:51.945270 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:51.945274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:51.945355 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:51.945438 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:51.946179 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:51.947125 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:51.948537 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:51.949028 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:51.949171 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:51.950410 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:51.950418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:51.950423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:51.950447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:51.950504 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:51.950522 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:51.950627 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:51.950835 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 20:58:51.950944 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 20:58:51.952567 21342 raft_consensus.cc:686] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [term 1 LEADER]: Becoming Leader. State: Replica: f602f9bc505c49f7b0a71a1cd05d3e13, State: Running, Role: LEADER
I20260923 20:58:51.953246 21342 consensus_queue.cc:237] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46387 } }
I20260923 20:58:51.953449 21332 sys_catalog.cc:564] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:51.956630 21343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46387 } } }
I20260923 20:58:51.956609 21344 sys_catalog.cc:455] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f602f9bc505c49f7b0a71a1cd05d3e13. Latest consensus state: current_term: 1 leader_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f602f9bc505c49f7b0a71a1cd05d3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46387 } } }
I20260923 20:58:51.957104 21343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:51.957115 21344 sys_catalog.cc:458] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:51.962848 21348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:51.967427 21348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:51.976689 21348 catalog_manager.cc:1347] Generated new cluster ID: 9317017836d0495ba2e5d1db370d86cd
I20260923 20:58:51.976828 21348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:51.986222 21348 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:51.987598 21348 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:51.997220 21348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f602f9bc505c49f7b0a71a1cd05d3e13: Generated new TSK 0
I20260923 20:58:51.998221 21348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:52.006475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21265
W20260923 20:58:52.007797 21327 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46425
W20260923 20:58:52.008634 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21265: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:58:52.008800 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21265
2026-09-23T20:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:52 dist-test-slave-plcg krb5kdc[21363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:52 dist-test-slave-plcg krb5kdc[21363](info): set up 2 sockets
Sep 23 20:58:52 dist-test-slave-plcg krb5kdc[21363](info): commencing operation
krb5kdc: starting...
W20260923 20:58:53.118263 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:53 dist-test-slave-plcg krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:53.385251 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:53.734977 21376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:53.735333 21376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:53.735442 21376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:53.744463 21376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:53.744619 21376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:53.744669 21376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:53.744725 21376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:53.744798 21376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:53.755355 21376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:53.756722 21376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:53.758101 21376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:53.771142 21381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:53.771142 21382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:53.771169 21384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:53.777556 21376 server_base.cc:1034] running on GCE node
I20260923 20:58:53.778815 21376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:53.780608 21376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:53.782063 21376 hybrid_clock.cc:648] HybridClock initialized: now 1790197133781966 us; error 94 us; skew 500 ppm
I20260923 20:58:53.782528 21376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:53 dist-test-slave-plcg krb5kdc[21363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:53.786190 21376 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:53.788161 21376 webserver.cc:460] Webserver started at http://127.0.0.1:46733/ using document root <none> and password file <none>
I20260923 20:58:53.789044 21376 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:53.789217 21376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:53.789595 21376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:53.794116 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "c685290135cf425a938d9e0023238fb9"
format_stamp: "Formatted at 2026-09-23 20:58:53 on dist-test-slave-plcg"
server_key: "f68c171e31ae9732f4d9ae2a3991242a"
server_key_iv: "7d290575cf45569cdf53c5c185e8fcab"
server_key_version: "encryptionkey@0"
I20260923 20:58:53.795395 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "c685290135cf425a938d9e0023238fb9"
format_stamp: "Formatted at 2026-09-23 20:58:53 on dist-test-slave-plcg"
server_key: "f68c171e31ae9732f4d9ae2a3991242a"
server_key_iv: "7d290575cf45569cdf53c5c185e8fcab"
server_key_version: "encryptionkey@0"
I20260923 20:58:53.802752 21376 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 20:58:53.806862 21391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:53.808336 21376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:53.808565 21376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "c685290135cf425a938d9e0023238fb9"
format_stamp: "Formatted at 2026-09-23 20:58:53 on dist-test-slave-plcg"
server_key: "f68c171e31ae9732f4d9ae2a3991242a"
server_key_iv: "7d290575cf45569cdf53c5c185e8fcab"
server_key_version: "encryptionkey@0"
I20260923 20:58:53.808815 21376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:53.815578 21376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:53.820374 21376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:53.820757 21376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:53.842794 21376 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41597
I20260923 20:58:53.842857 21442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41597 every 8 connection(s)
I20260923 20:58:53.844496 21376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:53.849124 21443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:53.853623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21376
I20260923 20:58:53.853955 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:53.854408 388 external_mini_cluster.cc:1428] Setting key dca63d341b84bd18def3840013bb0e00
I20260923 20:58:53.861724 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9: Bootstrap starting.
I20260923 20:58:53.865051 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:53.856337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47559 (local address 127.0.0.1:41597)
0923 20:58:53.857471 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:53.857490 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:53.857555 (+ 65us) server_negotiation.cc:407] Connection header received
0923 20:58:53.860274 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:53.860312 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:53.860878 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:53.861482 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:53.862924 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:53.862932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:53.862950 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:53.862966 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:53.863041 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:53.863064 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:53.863365 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:53.863547 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 20:58:53.863946 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":487,"threads_started":1}
I20260923 20:58:53.865906 21443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:53.867251 21443 log.cc:824] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:53.870402 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:53.870813 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9: No bootstrap required, opened a new log
I20260923 20:58:53.879146 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:53.875947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57077 (local address 127.0.0.1:41597)
0923 20:58:53.876223 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:53.876230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:53.876541 (+ 311us) server_negotiation.cc:407] Connection header received
0923 20:58:53.876759 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:53.876763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:53.876845 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:53.876962 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:53.878197 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:53.878202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:53.878205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:53.878227 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:53.878278 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:53.878301 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:53.878436 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:53.878616 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 20:58:53.878683 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 20:58:53.880453 21443 raft_consensus.cc:348] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685290135cf425a938d9e0023238fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260923 20:58:53.880937 21443 raft_consensus.cc:374] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:53.881049 21443 raft_consensus.cc:729] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c685290135cf425a938d9e0023238fb9, State: Initialized, Role: FOLLOWER
I20260923 20:58:53.881702 21443 consensus_queue.cc:260] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685290135cf425a938d9e0023238fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260923 20:58:53.882035 21443 raft_consensus.cc:388] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:53.882197 21443 raft_consensus.cc:482] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:53.882400 21443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:53.886566 21443 raft_consensus.cc:504] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685290135cf425a938d9e0023238fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260923 20:58:53.887328 21443 leader_election.cc:302] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c685290135cf425a938d9e0023238fb9; no voters:
I20260923 20:58:53.888743 21443 leader_election.cc:288] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:53.888890 21453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:53.889585 21453 raft_consensus.cc:686] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [term 1 LEADER]: Becoming Leader. State: Replica: c685290135cf425a938d9e0023238fb9, State: Running, Role: LEADER
I20260923 20:58:53.890434 21453 consensus_queue.cc:237] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685290135cf425a938d9e0023238fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260923 20:58:53.892572 21443 sys_catalog.cc:564] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:53.894510 21455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c685290135cf425a938d9e0023238fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685290135cf425a938d9e0023238fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } } }
I20260923 20:58:53.894405 21454 sys_catalog.cc:455] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c685290135cf425a938d9e0023238fb9. Latest consensus state: current_term: 1 leader_uuid: "c685290135cf425a938d9e0023238fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c685290135cf425a938d9e0023238fb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } } }
I20260923 20:58:53.895195 21455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:53.895191 21454 sys_catalog.cc:458] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:53.899528 21458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:53.905959 21458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:53.917529 21458 catalog_manager.cc:1347] Generated new cluster ID: 4c3d107c4add470697453155882fab85
I20260923 20:58:53.917671 21458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:53.931936 21458 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:53.932497 21458 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:53.942539 21458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c685290135cf425a938d9e0023238fb9: Generated new TSK 0
I20260923 20:58:53.943446 21458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:53.950244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21376
W20260923 20:58:53.951416 21438 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47559
W20260923 20:58:53.952263 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21376: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:58:53.952436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21376
2026-09-23T20:58:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:54 dist-test-slave-plcg krb5kdc[21474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:54 dist-test-slave-plcg krb5kdc[21474](info): set up 2 sockets
Sep 23 20:58:54 dist-test-slave-plcg krb5kdc[21474](info): commencing operation
krb5kdc: starting...
W20260923 20:58:55.062371 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:55 dist-test-slave-plcg krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:55.330883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:55.675709 21487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:55.676017 21487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:55.676205 21487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:55.683575 21487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:55.683698 21487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:55.683817 21487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:55.683924 21487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:55.683990 21487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:55.694103 21487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:55.695497 21487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:55.696873 21487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:55.709100 21492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:55.709013 21493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:55.709064 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:55.710064 21487 server_base.cc:1034] running on GCE node
I20260923 20:58:55.710966 21487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:55.712535 21487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:55.713757 21487 hybrid_clock.cc:648] HybridClock initialized: now 1790197135713714 us; error 34 us; skew 500 ppm
I20260923 20:58:55.714188 21487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:55 dist-test-slave-plcg krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:55.717552 21487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:55.719266 21487 webserver.cc:460] Webserver started at http://127.0.0.1:33547/ using document root <none> and password file <none>
I20260923 20:58:55.720017 21487 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:55.720171 21487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:55.720497 21487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:55.723706 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "412dde989dc44f18ae0805cb731df915"
format_stamp: "Formatted at 2026-09-23 20:58:55 on dist-test-slave-plcg"
server_key: "69e66c8c710949afd578ac94596f384a"
server_key_iv: "856867406351b44e44eaa2545ff5e536"
server_key_version: "encryptionkey@0"
I20260923 20:58:55.724556 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "412dde989dc44f18ae0805cb731df915"
format_stamp: "Formatted at 2026-09-23 20:58:55 on dist-test-slave-plcg"
server_key: "69e66c8c710949afd578ac94596f384a"
server_key_iv: "856867406351b44e44eaa2545ff5e536"
server_key_version: "encryptionkey@0"
I20260923 20:58:55.730293 21487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:58:55.734066 21502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:55.735536 21487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:58:55.735747 21487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "412dde989dc44f18ae0805cb731df915"
format_stamp: "Formatted at 2026-09-23 20:58:55 on dist-test-slave-plcg"
server_key: "69e66c8c710949afd578ac94596f384a"
server_key_iv: "856867406351b44e44eaa2545ff5e536"
server_key_version: "encryptionkey@0"
I20260923 20:58:55.735965 21487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:55.758754 21487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:55.763487 21487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:55.763811 21487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:55.783241 21487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40975
I20260923 20:58:55.783275 21553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40975 every 8 connection(s)
I20260923 20:58:55.784719 21487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:55.786989 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21487
I20260923 20:58:55.787197 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:55.787638 388 external_mini_cluster.cc:1428] Setting key 43cc46a65b236385ff5286be73451260
I20260923 20:58:55.789171 21554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:55.801537 21554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915: Bootstrap starting.
Sep 23 20:58:55 dist-test-slave-plcg krb5kdc[21474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:58:55.804939 21554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:55.806172 21554 log.cc:824] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:55.809175 21554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915: No bootstrap required, opened a new log
I20260923 20:58:55.810726 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:55.789480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:47613 (local address 127.0.0.1:40975)
0923 20:58:55.790358 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:55.790370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:55.790413 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:58:55.792099 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:55.792124 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:55.792573 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:55.793036 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:55.794597 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:55.795963 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:55.797335 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:55.798164 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:55.798400 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:55.802808 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:55.802817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:55.802834 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:55.802880 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:55.806292 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:55.807218 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:55.807226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:55.807233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:55.807416 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:55.807919 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:55.807930 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:55.807932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:55.808450 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:55.808775 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:55.809297 (+ 522us) server_negotiation.cc:299] Negotiation successful
0923 20:58:55.809631 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":374,"threads_started":1}
I20260923 20:58:55.814945 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:55.817859 21554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412dde989dc44f18ae0805cb731df915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40975 } }
I20260923 20:58:55.818357 21554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:55.818501 21554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412dde989dc44f18ae0805cb731df915, State: Initialized, Role: FOLLOWER
I20260923 20:58:55.819105 21554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412dde989dc44f18ae0805cb731df915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40975 } }
I20260923 20:58:55.819432 21554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:55.819620 21554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:55.819836 21554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:55.822738 21554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412dde989dc44f18ae0805cb731df915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40975 } }
I20260923 20:58:55.823391 21554 leader_election.cc:302] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412dde989dc44f18ae0805cb731df915; no voters:
I20260923 20:58:55.824209 21554 leader_election.cc:288] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:55.824357 21564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:55.826493 21554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:55.830169 21564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [term 1 LEADER]: Becoming Leader. State: Replica: 412dde989dc44f18ae0805cb731df915, State: Running, Role: LEADER
I20260923 20:58:55.831033 21564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412dde989dc44f18ae0805cb731df915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40975 } }
I20260923 20:58:55.834326 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412dde989dc44f18ae0805cb731df915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412dde989dc44f18ae0805cb731df915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40975 } } }
I20260923 20:58:55.834545 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412dde989dc44f18ae0805cb731df915. Latest consensus state: current_term: 1 leader_uuid: "412dde989dc44f18ae0805cb731df915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412dde989dc44f18ae0805cb731df915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40975 } } }
I20260923 20:58:55.834836 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:55.834843 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915 [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:55.842123 21574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:55.842741 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:55.819414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:59245 (local address 127.0.0.1:40975)
0923 20:58:55.819601 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:55.819612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:55.820104 (+ 492us) server_negotiation.cc:407] Connection header received
0923 20:58:55.820241 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:55.820250 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:55.820338 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:55.820455 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:55.821843 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:55.822983 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:55.824973 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:55.826034 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:55.826193 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:55.831206 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:55.831216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:55.831224 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:58:55.831291 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:55.837403 (+ 6112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:55.838914 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:55.838922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:55.838928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:55.839464 (+ 536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:58:55.840265 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:58:55.840273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:58:55.840279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:58:55.840964 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:55.841329 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:55.841974 (+ 645us) server_negotiation.cc:299] Negotiation successful
0923 20:58:55.842205 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 20:58:55.849472 21574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:55.859907 21574 catalog_manager.cc:1347] Generated new cluster ID: ef4f3ff74b034a14a35d9e3a8a337925
I20260923 20:58:55.860078 21574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:55.871712 21574 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:55.873319 21574 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:55.881976 21574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412dde989dc44f18ae0805cb731df915: Generated new TSK 0
I20260923 20:58:55.883101 21574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:55.900591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21487
I20260923 20:58:55.902026 21549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:58:55.995602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21487
W20260923 20:58:56.013355 21472 connection.cc:535] client connection to 127.0.0.1:40975 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40975): Connection reset by peer (error 104)
2026-09-23T20:58:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:56 dist-test-slave-plcg krb5kdc[21586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:56 dist-test-slave-plcg krb5kdc[21586](info): set up 2 sockets
Sep 23 20:58:56 dist-test-slave-plcg krb5kdc[21586](info): commencing operation
krb5kdc: starting...
W20260923 20:58:57.106580 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:57 dist-test-slave-plcg krb5kdc[21586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:57.379160 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:57.720760 21599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:57.721084 21599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:57.721179 21599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:57.728681 21599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:57.728811 21599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:57.728858 21599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:57.728922 21599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:57.728966 21599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:57.739565 21599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:57.740906 21599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:57.742239 21599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:57.754527 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:57.754562 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:57.754587 21607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:57.755431 21599 server_base.cc:1034] running on GCE node
I20260923 20:58:57.756337 21599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:57.757969 21599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:57.759178 21599 hybrid_clock.cc:648] HybridClock initialized: now 1790197137759119 us; error 58 us; skew 500 ppm
I20260923 20:58:57.759553 21599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:57 dist-test-slave-plcg krb5kdc[21586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:57.766930 21599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:57.768684 21599 webserver.cc:460] Webserver started at http://127.0.0.1:35513/ using document root <none> and password file <none>
I20260923 20:58:57.769433 21599 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:57.769587 21599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:57.769999 21599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:57.773144 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "0093cd73592f460d95bfe7f090e3ed8a"
format_stamp: "Formatted at 2026-09-23 20:58:57 on dist-test-slave-plcg"
server_key: "8937649ff67fbaeb170e58d32bd6fd4d"
server_key_iv: "5140165560253cc036f110ed34f5e3ad"
server_key_version: "encryptionkey@0"
I20260923 20:58:57.774052 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "0093cd73592f460d95bfe7f090e3ed8a"
format_stamp: "Formatted at 2026-09-23 20:58:57 on dist-test-slave-plcg"
server_key: "8937649ff67fbaeb170e58d32bd6fd4d"
server_key_iv: "5140165560253cc036f110ed34f5e3ad"
server_key_version: "encryptionkey@0"
I20260923 20:58:57.779633 21599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:58:57.783468 21614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:57.784979 21599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:58:57.785192 21599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "0093cd73592f460d95bfe7f090e3ed8a"
format_stamp: "Formatted at 2026-09-23 20:58:57 on dist-test-slave-plcg"
server_key: "8937649ff67fbaeb170e58d32bd6fd4d"
server_key_iv: "5140165560253cc036f110ed34f5e3ad"
server_key_version: "encryptionkey@0"
I20260923 20:58:57.785387 21599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:57.817637 21599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:57.822492 21599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:57.822826 21599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:57.842362 21599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35811
I20260923 20:58:57.842403 21665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35811 every 8 connection(s)
I20260923 20:58:57.843820 21599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:57.845345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21599
I20260923 20:58:57.845534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:57.846091 388 external_mini_cluster.cc:1428] Setting key a31d4eb5dc5590c13d2472f901fcd767
I20260923 20:58:57.850201 21666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:57.866430 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:57.848141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:54517 (local address 127.0.0.1:35811)
0923 20:58:57.849862 (+ 1721us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:57.849880 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:57.849935 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:58:57.853388 (+ 3453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:57.853426 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:57.854059 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:57.854576 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:57.856040 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:57.858004 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:57.860139 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:57.861055 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:57.861310 (+ 255us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:57.862746 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:57.862760 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:57.862785 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:57.863661 (+ 876us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:57.863771 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:57.863806 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:57.864016 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:57.864359 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 20:58:57.864589 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1417,"thread_start_us":1106,"threads_started":1}
I20260923 20:58:57.869482 21666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a: Bootstrap starting.
I20260923 20:58:57.871686 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:57.873924 21666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:57.875560 21666 log.cc:824] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:57.879706 21666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a: No bootstrap required, opened a new log
I20260923 20:58:57.885864 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:57.877511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:56167 (local address 127.0.0.1:35811)
0923 20:58:57.877700 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:57.877706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:57.878162 (+ 456us) server_negotiation.cc:407] Connection header received
0923 20:58:57.878320 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:57.878325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:57.878412 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:57.878530 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:57.879310 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:57.880672 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:57.882244 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:58:57.882975 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:58:57.883125 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:58:57.884824 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:57.884831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:57.884836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:57.884869 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:57.884928 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:57.884956 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:57.885053 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:57.885393 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 20:58:57.885456 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:58:57.890520 21666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093cd73592f460d95bfe7f090e3ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 20:58:57.891114 21666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:57.891260 21666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0093cd73592f460d95bfe7f090e3ed8a, State: Initialized, Role: FOLLOWER
I20260923 20:58:57.891978 21666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093cd73592f460d95bfe7f090e3ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 20:58:57.892278 21666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:57.892426 21666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:57.892596 21666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:57.895043 21666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093cd73592f460d95bfe7f090e3ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 20:58:57.895598 21666 leader_election.cc:302] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0093cd73592f460d95bfe7f090e3ed8a; no voters:
I20260923 20:58:57.896340 21666 leader_election.cc:288] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:57.896533 21676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:57.898129 21676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [term 1 LEADER]: Becoming Leader. State: Replica: 0093cd73592f460d95bfe7f090e3ed8a, State: Running, Role: LEADER
I20260923 20:58:57.898680 21666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:57.898819 21676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093cd73592f460d95bfe7f090e3ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } }
I20260923 20:58:57.902176 21678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0093cd73592f460d95bfe7f090e3ed8a. Latest consensus state: current_term: 1 leader_uuid: "0093cd73592f460d95bfe7f090e3ed8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093cd73592f460d95bfe7f090e3ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } } }
I20260923 20:58:57.902655 21678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:57.903182 21677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0093cd73592f460d95bfe7f090e3ed8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0093cd73592f460d95bfe7f090e3ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35811 } } }
I20260923 20:58:57.903493 21677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:57.903864 21684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:57.910979 21684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:57.923255 21684 catalog_manager.cc:1347] Generated new cluster ID: ed258e6a77a04e8c8de22bb1a2f865ed
I20260923 20:58:57.923406 21684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:57.940105 21684 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:57.941444 21684 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:57.951272 21684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0093cd73592f460d95bfe7f090e3ed8a: Generated new TSK 0
I20260923 20:58:57.952199 21684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:57.967556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21599
W20260923 20:58:57.968837 21661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.37:54517
W20260923 20:58:57.969662 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21599: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:58:57.969847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21599
W20260923 20:58:57.987190 21584 connection.cc:535] client connection to 127.0.0.1:35811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35811): Connection reset by peer (error 104)
2026-09-23T20:58:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:58:58 dist-test-slave-plcg krb5kdc[21697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:58:58 dist-test-slave-plcg krb5kdc[21697](info): set up 2 sockets
Sep 23 20:58:58 dist-test-slave-plcg krb5kdc[21697](info): commencing operation
krb5kdc: starting...
W20260923 20:58:59.082373 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:58:59 dist-test-slave-plcg krb5kdc[21697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:58:59.357693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:58:59.703824 21710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:58:59.704135 21710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:58:59.704232 21710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:58:59.711612 21710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:58:59.711745 21710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:58:59.711792 21710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:58:59.711855 21710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:58:59.711901 21710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:58:59.722098 21710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:58:59.723397 21710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:58:59.724777 21710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:58:59.737461 21715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:59.737435 21718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:58:59.737532 21716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:58:59.738489 21710 server_base.cc:1034] running on GCE node
I20260923 20:58:59.739430 21710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:58:59.740928 21710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:58:59.742175 21710 hybrid_clock.cc:648] HybridClock initialized: now 1790197139742125 us; error 47 us; skew 500 ppm
I20260923 20:58:59.742551 21710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:58:59 dist-test-slave-plcg krb5kdc[21697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:58:59.746019 21710 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:58:59.747727 21710 webserver.cc:460] Webserver started at http://127.0.0.1:36021/ using document root <none> and password file <none>
I20260923 20:58:59.748469 21710 fs_manager.cc:359] Metadata directory not provided
I20260923 20:58:59.748633 21710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:58:59.748970 21710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:58:59.752215 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "07d2c32916e4443c99617447a1cb4e1b"
format_stamp: "Formatted at 2026-09-23 20:58:59 on dist-test-slave-plcg"
server_key: "83630a6fd28adb9be50ed50f920e8916"
server_key_iv: "7a414a19a4379fb6db968d8c037fc4de"
server_key_version: "encryptionkey@0"
I20260923 20:58:59.753099 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "07d2c32916e4443c99617447a1cb4e1b"
format_stamp: "Formatted at 2026-09-23 20:58:59 on dist-test-slave-plcg"
server_key: "83630a6fd28adb9be50ed50f920e8916"
server_key_iv: "7a414a19a4379fb6db968d8c037fc4de"
server_key_version: "encryptionkey@0"
I20260923 20:58:59.758982 21710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:58:59.762972 21725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:58:59.764503 21710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:58:59.764736 21710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "07d2c32916e4443c99617447a1cb4e1b"
format_stamp: "Formatted at 2026-09-23 20:58:59 on dist-test-slave-plcg"
server_key: "83630a6fd28adb9be50ed50f920e8916"
server_key_iv: "7a414a19a4379fb6db968d8c037fc4de"
server_key_version: "encryptionkey@0"
I20260923 20:58:59.764977 21710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:58:59.771340 21710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:58:59.776207 21710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:58:59.776553 21710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:58:59.796591 21710 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33545
I20260923 20:58:59.796659 21776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33545 every 8 connection(s)
I20260923 20:58:59.798148 21710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:58:59.802549 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:58:59.805181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21710
I20260923 20:58:59.805362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:58:59.805869 388 external_mini_cluster.cc:1428] Setting key a9492045f8a0f1b1cf24ff25b824a33c
I20260923 20:58:59.814280 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:59.807766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:35129 (local address 127.0.0.1:33545)
0923 20:58:59.808597 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:59.808618 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:59.808665 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:58:59.810524 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:59.810549 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:59.810934 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:59.811376 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:59.812476 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:59.812482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:59.812499 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:59.812517 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:59.812575 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:59.812597 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:59.812865 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:59.813024 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 20:58:59.813227 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":345,"threads_started":1}
I20260923 20:58:59.815402 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b: Bootstrap starting.
I20260923 20:58:59.818630 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:58:59.819202 21777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b: Neither blocks nor log segments found. Creating new log.
I20260923 20:58:59.820295 21777 log.cc:824] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:58:59.823678 21777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b: No bootstrap required, opened a new log
I20260923 20:58:59.826812 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:58:59.824088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:34619 (local address 127.0.0.1:33545)
0923 20:58:59.824310 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:58:59.824316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:58:59.824794 (+ 478us) server_negotiation.cc:407] Connection header received
0923 20:58:59.824943 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:58:59.824948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:58:59.825019 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:58:59.825148 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:58:59.826113 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:58:59.826118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:58:59.826123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:58:59.826152 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:58:59.826211 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:58:59.826233 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:58:59.826366 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:58:59.826426 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 20:58:59.826491 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:58:59.832918 21777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d2c32916e4443c99617447a1cb4e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260923 20:58:59.833333 21777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:58:59.833408 21777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d2c32916e4443c99617447a1cb4e1b, State: Initialized, Role: FOLLOWER
I20260923 20:58:59.834029 21777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d2c32916e4443c99617447a1cb4e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260923 20:58:59.834280 21777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:58:59.834441 21777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:58:59.834651 21777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:58:59.836971 21777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d2c32916e4443c99617447a1cb4e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260923 20:58:59.837510 21777 leader_election.cc:302] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07d2c32916e4443c99617447a1cb4e1b; no voters:
I20260923 20:58:59.838227 21777 leader_election.cc:288] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:58:59.838378 21787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:58:59.839169 21787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [term 1 LEADER]: Becoming Leader. State: Replica: 07d2c32916e4443c99617447a1cb4e1b, State: Running, Role: LEADER
I20260923 20:58:59.839725 21787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d2c32916e4443c99617447a1cb4e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } }
I20260923 20:58:59.840432 21777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:58:59.842931 21789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07d2c32916e4443c99617447a1cb4e1b. Latest consensus state: current_term: 1 leader_uuid: "07d2c32916e4443c99617447a1cb4e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d2c32916e4443c99617447a1cb4e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } } }
I20260923 20:58:59.843466 21789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:59.843993 21788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07d2c32916e4443c99617447a1cb4e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d2c32916e4443c99617447a1cb4e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33545 } } }
I20260923 20:58:59.844373 21788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b [sys.catalog]: This master's current role is: LEADER
I20260923 20:58:59.846052 21794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:58:59.851625 21794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:58:59.863381 21794 catalog_manager.cc:1347] Generated new cluster ID: 75ba56a243774054add80c66ad54d373
I20260923 20:58:59.863529 21794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:58:59.899416 21794 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:58:59.900020 21794 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:58:59.911098 21794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07d2c32916e4443c99617447a1cb4e1b: Generated new TSK 0
I20260923 20:58:59.912302 21794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:58:59.928664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21710
W20260923 20:58:59.930790 21772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.37:35129
W20260923 20:58:59.931658 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21710: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:58:59.931815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21710
2026-09-23T20:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:00 dist-test-slave-plcg krb5kdc[21808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:00 dist-test-slave-plcg krb5kdc[21808](info): set up 2 sockets
Sep 23 20:59:00 dist-test-slave-plcg krb5kdc[21808](info): commencing operation
krb5kdc: starting...
W20260923 20:59:01.059044 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:01 dist-test-slave-plcg krb5kdc[21808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:59:01.341018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:01.695444 21821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:01.695847 21821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:01.696026 21821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:01.703660 21821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:01.703835 21821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:01.703953 21821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:01.704066 21821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:01.704146 21821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:01.714401 21821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:01.715832 21821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:01.717200 21821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:01.729605 21827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:01.729830 21826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:01.729830 21829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:01.730484 21821 server_base.cc:1034] running on GCE node
I20260923 20:59:01.731314 21821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:01.732817 21821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:01.734056 21821 hybrid_clock.cc:648] HybridClock initialized: now 1790197141734012 us; error 52 us; skew 500 ppm
I20260923 20:59:01.734454 21821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:01 dist-test-slave-plcg krb5kdc[21808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:01.737916 21821 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:59:01.739471 21821 webserver.cc:460] Webserver started at http://127.0.0.1:33791/ using document root <none> and password file <none>
I20260923 20:59:01.740222 21821 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:01.740378 21821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:01.740708 21821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:01.743904 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "1bc1d3f5bab942bab0a62d76fabf8461"
format_stamp: "Formatted at 2026-09-23 20:59:01 on dist-test-slave-plcg"
server_key: "e1dcba9ad4760e39485bbef6bdb08577"
server_key_iv: "457a120511b2dd2b673cb55fca8dcd7a"
server_key_version: "encryptionkey@0"
I20260923 20:59:01.744758 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "1bc1d3f5bab942bab0a62d76fabf8461"
format_stamp: "Formatted at 2026-09-23 20:59:01 on dist-test-slave-plcg"
server_key: "e1dcba9ad4760e39485bbef6bdb08577"
server_key_iv: "457a120511b2dd2b673cb55fca8dcd7a"
server_key_version: "encryptionkey@0"
I20260923 20:59:01.750670 21821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260923 20:59:01.754599 21836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:01.756047 21821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:59:01.756295 21821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "1bc1d3f5bab942bab0a62d76fabf8461"
format_stamp: "Formatted at 2026-09-23 20:59:01 on dist-test-slave-plcg"
server_key: "e1dcba9ad4760e39485bbef6bdb08577"
server_key_iv: "457a120511b2dd2b673cb55fca8dcd7a"
server_key_version: "encryptionkey@0"
I20260923 20:59:01.756539 21821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:01.791676 21821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:01.795957 21821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:01.796304 21821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:01.816149 21821 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41969
I20260923 20:59:01.816174 21887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41969 every 8 connection(s)
I20260923 20:59:01.817653 21821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:01.818774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21821
I20260923 20:59:01.818972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:01.819481 388 external_mini_cluster.cc:1428] Setting key cbf690b0fe5c2413627194dc979aaf5d
I20260923 20:59:01.822583 21888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:59:01 dist-test-slave-plcg krb5kdc[21808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:59:01.836796 21888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461: Bootstrap starting.
I20260923 20:59:01.841418 21888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:01.842821 21888 log.cc:824] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:01.846362 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:01.821500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:35633 (local address 127.0.0.1:41969)
0923 20:59:01.822410 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:01.822422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:01.822475 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:59:01.824357 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:01.824382 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:01.824851 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:01.825241 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:01.827133 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:01.828714 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:01.830713 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:01.831469 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:01.831628 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:01.836763 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:01.836782 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:01.836806 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:01.836869 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:01.840918 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:01.842050 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:01.842058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:01.842067 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:01.842283 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:01.842788 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:01.842795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:01.842800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:01.843576 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:01.843890 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:01.844560 (+ 670us) server_negotiation.cc:299] Negotiation successful
0923 20:59:01.844947 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":408,"threads_started":1}
I20260923 20:59:01.846730 21888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461: No bootstrap required, opened a new log
I20260923 20:59:01.851541 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:01.856168 21888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260923 20:59:01.856830 21888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:01.856959 21888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc1d3f5bab942bab0a62d76fabf8461, State: Initialized, Role: FOLLOWER
I20260923 20:59:01.857766 21888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260923 20:59:01.858129 21888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:01.858297 21888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:01.858491 21888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:01.861773 21888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260923 20:59:01.862555 21888 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc1d3f5bab942bab0a62d76fabf8461; no voters:
I20260923 20:59:01.863358 21888 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:01.863481 21898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:01.866019 21888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:01.866053 21898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc1d3f5bab942bab0a62d76fabf8461, State: Running, Role: LEADER
I20260923 20:59:01.866845 21898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260923 20:59:01.870837 21899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } } }
I20260923 20:59:01.871368 21899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:01.872089 21900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc1d3f5bab942bab0a62d76fabf8461. Latest consensus state: current_term: 1 leader_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc1d3f5bab942bab0a62d76fabf8461" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } } }
I20260923 20:59:01.872442 21900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:01.872500 21907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:01.878877 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:01.856200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:39011 (local address 127.0.0.1:41969)
0923 20:59:01.856424 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:01.856434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:01.857016 (+ 582us) server_negotiation.cc:407] Connection header received
0923 20:59:01.857264 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:01.857269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:01.857366 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:01.857483 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:01.859241 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:01.860516 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:01.862247 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:01.862898 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:01.863082 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:01.866863 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:01.866870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:01.866875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:01.866928 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:01.874443 (+ 7515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:01.875398 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:01.875403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:01.875407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:01.875616 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:01.876265 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:01.876271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:01.876274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:01.876694 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:01.877050 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:01.878279 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0923 20:59:01.878481 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:59:01.879653 21907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:01.895562 21907 catalog_manager.cc:1347] Generated new cluster ID: 5e17d206416e4d3dbeaab4b89482a1e0
I20260923 20:59:01.895699 21907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:01.917977 21907 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:01.919891 21907 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:01.929076 21907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc1d3f5bab942bab0a62d76fabf8461: Generated new TSK 0
I20260923 20:59:01.930102 21907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:01.941538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21821
I20260923 20:59:01.942940 21883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:02.037575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21821
W20260923 20:59:02.055343 21806 connection.cc:535] client connection to 127.0.0.1:41969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41969): Connection reset by peer (error 104)
2026-09-23T20:59:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2110 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:02 dist-test-slave-plcg krb5kdc[21920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:02 dist-test-slave-plcg krb5kdc[21920](info): set up 2 sockets
Sep 23 20:59:02 dist-test-slave-plcg krb5kdc[21920](info): commencing operation
krb5kdc: starting...
W20260923 20:59:03.142452 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:03 dist-test-slave-plcg krb5kdc[21920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:59:03.417177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:03.765219 21933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:03.765545 21933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:03.765724 21933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:03.773434 21933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:03.773573 21933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:03.773674 21933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:03.773782 21933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:03.773901 21933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:03.784837 21933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:03.786340 21933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:03.787724 21933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:03.800890 21938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:03.800751 21939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:03.800751 21941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:03.801927 21933 server_base.cc:1034] running on GCE node
I20260923 20:59:03.802870 21933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:03.804435 21933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:03.805721 21933 hybrid_clock.cc:648] HybridClock initialized: now 1790197143805669 us; error 41 us; skew 500 ppm
I20260923 20:59:03.806205 21933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:03 dist-test-slave-plcg krb5kdc[21920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:03.813505 21933 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:59:03.815414 21933 webserver.cc:460] Webserver started at http://127.0.0.1:35375/ using document root <none> and password file <none>
I20260923 20:59:03.816203 21933 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:03.816385 21933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:03.816712 21933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:03.819994 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "dda06b8d61674e3a9600455bc348b5d8"
format_stamp: "Formatted at 2026-09-23 20:59:03 on dist-test-slave-plcg"
server_key: "aa3ea50161ae944cfa3b7b2d17a30f9f"
server_key_iv: "35c34f7fe84ef6a7b8e391d8f83d037e"
server_key_version: "encryptionkey@0"
I20260923 20:59:03.820865 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "dda06b8d61674e3a9600455bc348b5d8"
format_stamp: "Formatted at 2026-09-23 20:59:03 on dist-test-slave-plcg"
server_key: "aa3ea50161ae944cfa3b7b2d17a30f9f"
server_key_iv: "35c34f7fe84ef6a7b8e391d8f83d037e"
server_key_version: "encryptionkey@0"
I20260923 20:59:03.826676 21933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:59:03.830714 21948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:03.832214 21933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:59:03.832427 21933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "dda06b8d61674e3a9600455bc348b5d8"
format_stamp: "Formatted at 2026-09-23 20:59:03 on dist-test-slave-plcg"
server_key: "aa3ea50161ae944cfa3b7b2d17a30f9f"
server_key_iv: "35c34f7fe84ef6a7b8e391d8f83d037e"
server_key_version: "encryptionkey@0"
I20260923 20:59:03.832653 21933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:03.846022 21933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:03.850565 21933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:03.850878 21933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:03.870631 21933 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45393
I20260923 20:59:03.870668 21999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45393 every 8 connection(s)
I20260923 20:59:03.872152 21933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:03.873781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 21933
I20260923 20:59:03.873987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:03.874563 388 external_mini_cluster.cc:1428] Setting key 80148f2b4b84be66d01151073d8925b5
I20260923 20:59:03.877069 22000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:03.893203 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:03.876524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:39989 (local address 127.0.0.1:45393)
0923 20:59:03.877580 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:03.877605 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:03.877907 (+ 302us) server_negotiation.cc:407] Connection header received
0923 20:59:03.879800 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:03.879824 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:03.880274 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:03.880660 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:03.881260 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:03.883225 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:03.885779 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:03.887228 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:03.887440 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:03.889391 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:03.889404 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:03.889433 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:03.890323 (+ 890us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:03.890450 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:03.890492 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:03.890700 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:03.891356 (+ 656us) server_negotiation.cc:299] Negotiation successful
0923 20:59:03.891602 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":430,"threads_started":1}
I20260923 20:59:03.894083 22000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8: Bootstrap starting.
I20260923 20:59:03.897671 22000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:03.897958 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:03.899133 22000 log.cc:824] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:03.903057 22000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8: No bootstrap required, opened a new log
I20260923 20:59:03.910701 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:03.903245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:33159 (local address 127.0.0.1:45393)
0923 20:59:03.903465 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:03.903475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:03.903992 (+ 517us) server_negotiation.cc:407] Connection header received
0923 20:59:03.904200 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:03.904204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:03.904268 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:03.904355 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:03.904942 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:03.905749 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:03.907805 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:03.908276 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:03.908380 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:03.909774 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:03.909778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:03.909782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:03.909802 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:03.909916 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:03.909934 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:03.910030 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:03.910252 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 20:59:03.910306 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:59:03.913249 22000 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda06b8d61674e3a9600455bc348b5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260923 20:59:03.913712 22000 raft_consensus.cc:374] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:03.913879 22000 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda06b8d61674e3a9600455bc348b5d8, State: Initialized, Role: FOLLOWER
I20260923 20:59:03.914511 22000 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda06b8d61674e3a9600455bc348b5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260923 20:59:03.914788 22000 raft_consensus.cc:388] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:03.914938 22000 raft_consensus.cc:482] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:03.915129 22000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:03.917443 22000 raft_consensus.cc:504] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda06b8d61674e3a9600455bc348b5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260923 20:59:03.918092 22000 leader_election.cc:302] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dda06b8d61674e3a9600455bc348b5d8; no voters:
I20260923 20:59:03.918704 22000 leader_election.cc:288] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:03.918957 22010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:03.919673 22010 raft_consensus.cc:686] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [term 1 LEADER]: Becoming Leader. State: Replica: dda06b8d61674e3a9600455bc348b5d8, State: Running, Role: LEADER
I20260923 20:59:03.920262 22010 consensus_queue.cc:237] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda06b8d61674e3a9600455bc348b5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260923 20:59:03.920955 22000 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:03.923461 22012 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dda06b8d61674e3a9600455bc348b5d8. Latest consensus state: current_term: 1 leader_uuid: "dda06b8d61674e3a9600455bc348b5d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda06b8d61674e3a9600455bc348b5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } } }
I20260923 20:59:03.923892 22012 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:03.925256 22015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:03.924845 22011 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dda06b8d61674e3a9600455bc348b5d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda06b8d61674e3a9600455bc348b5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } } }
I20260923 20:59:03.925552 22011 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:03.932832 22015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:03.944876 22015 catalog_manager.cc:1347] Generated new cluster ID: cd3257c3e37b458a857062383427ea42
I20260923 20:59:03.945020 22015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:03.961767 22015 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:03.963769 22015 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:03.974042 22015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dda06b8d61674e3a9600455bc348b5d8: Generated new TSK 0
I20260923 20:59:03.975275 22015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:03.985440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21933
W20260923 20:59:03.987003 21995 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.37:39989
W20260923 20:59:03.987857 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 21933: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:59:03.988025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 21933
W20260923 20:59:04.002952 21918 connection.cc:535] client connection to 127.0.0.1:45393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45393): Connection reset by peer (error 104)
2026-09-23T20:59:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:04 dist-test-slave-plcg krb5kdc[22031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:04 dist-test-slave-plcg krb5kdc[22031](info): set up 2 sockets
Sep 23 20:59:04 dist-test-slave-plcg krb5kdc[22031](info): commencing operation
krb5kdc: starting...
W20260923 20:59:05.098316 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:05 dist-test-slave-plcg krb5kdc[22031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:59:05.374434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:05.728672 22044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:05.729000 22044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:05.729197 22044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:05.736861 22044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:05.737002 22044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:05.737100 22044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:05.737195 22044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:05.737313 22044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:05.747740 22044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:05.749184 22044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:05.750648 22044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:05.764104 22052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:05.764199 22049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:05.764081 22050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:05.764380 22044 server_base.cc:1034] running on GCE node
I20260923 20:59:05.765533 22044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:05.767091 22044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:05.768359 22044 hybrid_clock.cc:648] HybridClock initialized: now 1790197145768282 us; error 56 us; skew 500 ppm
I20260923 20:59:05.768747 22044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:05 dist-test-slave-plcg krb5kdc[22031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:05.772462 22044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:59:05.774188 22044 webserver.cc:460] Webserver started at http://127.0.0.1:33421/ using document root <none> and password file <none>
I20260923 20:59:05.774953 22044 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:05.775082 22044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:05.775370 22044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:05.778729 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "0c423a2a98384f8bbde5fe228ce308f0"
format_stamp: "Formatted at 2026-09-23 20:59:05 on dist-test-slave-plcg"
server_key: "b9c80bcd1963c05ba8b74d9f6254b269"
server_key_iv: "401f69670ec91b31ca0f19966865f1a0"
server_key_version: "encryptionkey@0"
I20260923 20:59:05.779641 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c423a2a98384f8bbde5fe228ce308f0"
format_stamp: "Formatted at 2026-09-23 20:59:05 on dist-test-slave-plcg"
server_key: "b9c80bcd1963c05ba8b74d9f6254b269"
server_key_iv: "401f69670ec91b31ca0f19966865f1a0"
server_key_version: "encryptionkey@0"
I20260923 20:59:05.785455 22044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:59:05.789578 22059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:05.791155 22044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:59:05.791394 22044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "0c423a2a98384f8bbde5fe228ce308f0"
format_stamp: "Formatted at 2026-09-23 20:59:05 on dist-test-slave-plcg"
server_key: "b9c80bcd1963c05ba8b74d9f6254b269"
server_key_iv: "401f69670ec91b31ca0f19966865f1a0"
server_key_version: "encryptionkey@0"
I20260923 20:59:05.791633 22044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:05.798026 22044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:05.802992 22044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:05.803352 22044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:05.823140 22044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43407
I20260923 20:59:05.823184 22110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43407 every 8 connection(s)
I20260923 20:59:05.824753 22044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:05.829003 22111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:05.830945 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22044
I20260923 20:59:05.831153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:05.831614 388 external_mini_cluster.cc:1428] Setting key 93e221e73349ea71829d67b5487e9843
I20260923 20:59:05.840387 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:05.833614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:37009 (local address 127.0.0.1:43407)
0923 20:59:05.834594 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:05.834607 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:05.834656 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:59:05.836440 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:05.836466 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:05.836845 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:05.837197 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:05.838515 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:05.838522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:05.838536 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:05.838554 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:05.838632 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:05.838659 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:05.838878 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:05.838990 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 20:59:05.839269 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":474,"threads_started":1}
I20260923 20:59:05.841941 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0: Bootstrap starting.
I20260923 20:59:05.844623 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:05.845594 22111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:05.846764 22111 log.cc:824] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:05.850715 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0: No bootstrap required, opened a new log
I20260923 20:59:05.853076 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:05.849502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:58925 (local address 127.0.0.1:43407)
0923 20:59:05.849731 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:05.849741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:05.850416 (+ 675us) server_negotiation.cc:407] Connection header received
0923 20:59:05.850580 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:05.850585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:05.850649 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:05.850787 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:05.851916 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:05.851922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:05.851927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:05.851948 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:05.852004 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:05.852025 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:05.852158 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:05.852494 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 20:59:05.852629 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:59:05.860435 22111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c423a2a98384f8bbde5fe228ce308f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
I20260923 20:59:05.861048 22111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:05.861194 22111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c423a2a98384f8bbde5fe228ce308f0, State: Initialized, Role: FOLLOWER
I20260923 20:59:05.861933 22111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c423a2a98384f8bbde5fe228ce308f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
I20260923 20:59:05.862216 22111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:05.862380 22111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:05.862571 22111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:05.864985 22111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c423a2a98384f8bbde5fe228ce308f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
I20260923 20:59:05.866206 22111 leader_election.cc:302] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c423a2a98384f8bbde5fe228ce308f0; no voters:
I20260923 20:59:05.867152 22111 leader_election.cc:288] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:05.867230 22121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:05.869004 22121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [term 1 LEADER]: Becoming Leader. State: Replica: 0c423a2a98384f8bbde5fe228ce308f0, State: Running, Role: LEADER
I20260923 20:59:05.869932 22121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c423a2a98384f8bbde5fe228ce308f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } }
I20260923 20:59:05.870463 22111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:05.873760 22123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c423a2a98384f8bbde5fe228ce308f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c423a2a98384f8bbde5fe228ce308f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } } }
I20260923 20:59:05.874329 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c423a2a98384f8bbde5fe228ce308f0. Latest consensus state: current_term: 1 leader_uuid: "0c423a2a98384f8bbde5fe228ce308f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c423a2a98384f8bbde5fe228ce308f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43407 } } }
I20260923 20:59:05.874696 22123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:05.874713 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:05.876753 22130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:05.890072 22130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:05.894243 22138 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:59:05.903947 22130 catalog_manager.cc:1347] Generated new cluster ID: 2c016a1e25bf44aa9acb3311fa9ef6d2
I20260923 20:59:05.904119 22130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:05.937940 22130 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:05.938555 22130 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:05.954581 22130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c423a2a98384f8bbde5fe228ce308f0: Generated new TSK 0
I20260923 20:59:05.955802 22130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:05.970122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22044
W20260923 20:59:05.971619 22106 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.37:37009
W20260923 20:59:05.973217 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22044: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:59:05.973409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22044
2026-09-23T20:59:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:06 dist-test-slave-plcg krb5kdc[22142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:06 dist-test-slave-plcg krb5kdc[22142](info): set up 2 sockets
Sep 23 20:59:06 dist-test-slave-plcg krb5kdc[22142](info): commencing operation
krb5kdc: starting...
W20260923 20:59:07.114305 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:07 dist-test-slave-plcg krb5kdc[22142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:07.393219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46813
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:07.746915 22155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:07.747272 22155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:07.747370 22155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:07.754911 22155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:07.755043 22155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:07.755095 22155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:07.755149 22155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:07.755194 22155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:07.766326 22155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:07.767637 22155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:07.768970 22155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:07.781522 22163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:07.781548 22161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:07.781775 22160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:07.782582 22155 server_base.cc:1034] running on GCE node
I20260923 20:59:07.783412 22155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:07.784935 22155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:07.786187 22155 hybrid_clock.cc:648] HybridClock initialized: now 1790197147786130 us; error 50 us; skew 500 ppm
I20260923 20:59:07.786629 22155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:07 dist-test-slave-plcg krb5kdc[22142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:07.790489 22155 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:07.792194 22155 webserver.cc:460] Webserver started at http://127.0.97.62:34333/ using document root <none> and password file <none>
I20260923 20:59:07.792975 22155 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:07.793128 22155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:07.793475 22155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:07.796788 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "25cbad143b4b4a28976f133268b23b4c"
format_stamp: "Formatted at 2026-09-23 20:59:07 on dist-test-slave-plcg"
server_key: "30b6a00b78417657fac3c5ec4ceb3d77"
server_key_iv: "641f83d87507e4d4f5f86f1ef21faa2a"
server_key_version: "encryptionkey@0"
I20260923 20:59:07.797645 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "25cbad143b4b4a28976f133268b23b4c"
format_stamp: "Formatted at 2026-09-23 20:59:07 on dist-test-slave-plcg"
server_key: "30b6a00b78417657fac3c5ec4ceb3d77"
server_key_iv: "641f83d87507e4d4f5f86f1ef21faa2a"
server_key_version: "encryptionkey@0"
I20260923 20:59:07.803336 22155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:59:07.807230 22170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:07.808890 22155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:59:07.809207 22155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "25cbad143b4b4a28976f133268b23b4c"
format_stamp: "Formatted at 2026-09-23 20:59:07 on dist-test-slave-plcg"
server_key: "30b6a00b78417657fac3c5ec4ceb3d77"
server_key_iv: "641f83d87507e4d4f5f86f1ef21faa2a"
server_key_version: "encryptionkey@0"
I20260923 20:59:07.809448 22155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:07.828495 22155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:07.833411 22155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:07.833745 22155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:07.854908 22155 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46813
I20260923 20:59:07.854943 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46813 every 8 connection(s)
I20260923 20:59:07.856398 22155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:07.860484 22222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:07.861662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22155
I20260923 20:59:07.861917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:07.862404 388 external_mini_cluster.cc:1428] Setting key 1a9c8a21526b5c7dd0e9efc666c1175d
I20260923 20:59:07.873940 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c: Bootstrap starting.
Sep 23 20:59:07 dist-test-slave-plcg krb5kdc[22142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:59:07.877557 22222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:07.878803 22222 log.cc:824] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:07.882018 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c: No bootstrap required, opened a new log
I20260923 20:59:07.888345 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:07.864267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36079 (local address 127.0.97.62:46813)
0923 20:59:07.865080 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:07.865097 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:07.865140 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:59:07.867143 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:07.867172 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:07.867696 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:07.868151 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:07.869706 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:07.871138 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:07.872504 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:07.873261 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:07.873476 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:07.878355 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:07.878364 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:07.878382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:07.878432 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:07.881707 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:07.882712 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:07.882724 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:07.882734 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:07.883027 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:07.883739 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:07.883744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:07.883748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:07.884734 (+ 986us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:07.885121 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:07.885996 (+ 875us) server_negotiation.cc:299] Negotiation successful
0923 20:59:07.886603 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260923 20:59:07.893338 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:07.894132 22222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cbad143b4b4a28976f133268b23b4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46813 } }
I20260923 20:59:07.894567 22222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:07.894685 22222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cbad143b4b4a28976f133268b23b4c, State: Initialized, Role: FOLLOWER
I20260923 20:59:07.895313 22222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cbad143b4b4a28976f133268b23b4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46813 } }
I20260923 20:59:07.895568 22222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:07.895732 22222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:07.895895 22222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:07.898852 22222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cbad143b4b4a28976f133268b23b4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46813 } }
I20260923 20:59:07.899603 22222 leader_election.cc:302] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25cbad143b4b4a28976f133268b23b4c; no voters:
I20260923 20:59:07.900344 22222 leader_election.cc:288] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:07.900485 22232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:07.902064 22232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [term 1 LEADER]: Becoming Leader. State: Replica: 25cbad143b4b4a28976f133268b23b4c, State: Running, Role: LEADER
I20260923 20:59:07.902766 22232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cbad143b4b4a28976f133268b23b4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46813 } }
I20260923 20:59:07.902902 22222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:07.905694 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25cbad143b4b4a28976f133268b23b4c. Latest consensus state: current_term: 1 leader_uuid: "25cbad143b4b4a28976f133268b23b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cbad143b4b4a28976f133268b23b4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46813 } } }
I20260923 20:59:07.906188 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:07.906786 22233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25cbad143b4b4a28976f133268b23b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25cbad143b4b4a28976f133268b23b4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46813 } } }
I20260923 20:59:07.907083 22233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:07.910338 22238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:07.918071 22238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:07.922703 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:07.898246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35317 (local address 127.0.97.62:46813)
0923 20:59:07.898460 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:07.898470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:07.899043 (+ 573us) server_negotiation.cc:407] Connection header received
0923 20:59:07.899225 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:07.899229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:07.899295 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:07.899384 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:07.901491 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:07.902755 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:07.904949 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:07.906056 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:07.906196 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:07.911088 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:07.911098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:07.911104 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:07.911175 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:07.915865 (+ 4690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:07.917061 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:07.917066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:07.917070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:07.917243 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:07.917842 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:07.917850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:07.917853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:07.918347 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:07.921856 (+ 3509us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:07.922077 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 20:59:07.922291 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:59:07.933866 22238 catalog_manager.cc:1347] Generated new cluster ID: e44293b347fc46d3bb6d9d95612e404c
I20260923 20:59:07.934170 22238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:07.974303 22238 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:07.976536 22238 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:07.992108 22238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25cbad143b4b4a28976f133268b23b4c: Generated new TSK 0
I20260923 20:59:07.993417 22238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:08.007170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22155
I20260923 20:59:08.008581 22217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:08.106184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22155
W20260923 20:59:08.124322 22140 connection.cc:535] client connection to 127.0.97.62:46813 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:46813): Connection reset by peer (error 104)
2026-09-23T20:59:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:08 dist-test-slave-plcg krb5kdc[22254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:08 dist-test-slave-plcg krb5kdc[22254](info): set up 2 sockets
Sep 23 20:59:08 dist-test-slave-plcg krb5kdc[22254](info): commencing operation
krb5kdc: starting...
W20260923 20:59:09.226317 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:09 dist-test-slave-plcg krb5kdc[22254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:09.498163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38407
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:09.848158 22267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:09.848562 22267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:09.848661 22267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:09.856861 22267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:09.857026 22267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:09.857081 22267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:09.857165 22267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:09.857209 22267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:09.867766 22267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:09.869140 22267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:09.870525 22267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:09.883823 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:09.883982 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:09.884549 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:09.885357 22267 server_base.cc:1034] running on GCE node
I20260923 20:59:09.886351 22267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:09.887967 22267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:09.889564 22267 hybrid_clock.cc:648] HybridClock initialized: now 1790197149889496 us; error 52 us; skew 500 ppm
I20260923 20:59:09.890121 22267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:09 dist-test-slave-plcg krb5kdc[22254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:09.897710 22267 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:09.899737 22267 webserver.cc:460] Webserver started at http://127.0.97.62:35877/ using document root <none> and password file <none>
I20260923 20:59:09.900508 22267 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:09.900627 22267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:09.900905 22267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:09.904245 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "6eee576fda854eaba29e51a70a3fb179"
format_stamp: "Formatted at 2026-09-23 20:59:09 on dist-test-slave-plcg"
server_key: "3604120671323e22895f68bd0ffffb7e"
server_key_iv: "abddf899497c09f0ffe782a5862330ab"
server_key_version: "encryptionkey@0"
I20260923 20:59:09.905066 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "6eee576fda854eaba29e51a70a3fb179"
format_stamp: "Formatted at 2026-09-23 20:59:09 on dist-test-slave-plcg"
server_key: "3604120671323e22895f68bd0ffffb7e"
server_key_iv: "abddf899497c09f0ffe782a5862330ab"
server_key_version: "encryptionkey@0"
I20260923 20:59:09.910727 22267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:59:09.914572 22282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:09.916266 22267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:59:09.916445 22267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "6eee576fda854eaba29e51a70a3fb179"
format_stamp: "Formatted at 2026-09-23 20:59:09 on dist-test-slave-plcg"
server_key: "3604120671323e22895f68bd0ffffb7e"
server_key_iv: "abddf899497c09f0ffe782a5862330ab"
server_key_version: "encryptionkey@0"
I20260923 20:59:09.916658 22267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:09.935309 22267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:09.940140 22267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:09.940390 22267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:09.960578 22267 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38407
I20260923 20:59:09.960614 22333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38407 every 8 connection(s)
I20260923 20:59:09.962163 22267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:09.965596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22267
I20260923 20:59:09.965777 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:09.966243 388 external_mini_cluster.cc:1428] Setting key 1c2e382c5b181408a375429725d5d154
I20260923 20:59:09.966344 22334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:09.980129 22334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179: Bootstrap starting.
I20260923 20:59:09.980939 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:09.968085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38109 (local address 127.0.97.62:38407)
0923 20:59:09.968906 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:09.968919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:09.968963 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:59:09.971294 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:09.971376 (+ 82us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:09.971913 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:09.972398 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:09.973014 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:09.974493 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:09.975917 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:09.976718 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:09.976948 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:09.978444 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:09.978452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:09.978470 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:09.979060 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:09.979156 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:09.979183 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:09.979363 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:09.979654 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 20:59:09.979847 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260923 20:59:09.983986 22334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:09.985283 22334 log.cc:824] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:09.985386 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:09.988442 22334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179: No bootstrap required, opened a new log
I20260923 20:59:09.998945 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:09.991206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58207 (local address 127.0.97.62:38407)
0923 20:59:09.991450 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:09.991459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:09.991982 (+ 523us) server_negotiation.cc:407] Connection header received
0923 20:59:09.992201 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:09.992206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:09.992278 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:09.992368 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:09.992962 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:09.993882 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:09.995926 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:09.996616 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:09.996744 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:09.998091 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:09.998096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:09.998099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:09.998119 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:09.998170 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:09.998186 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:09.998273 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:09.998510 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 20:59:09.998586 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:59:09.998911 22334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eee576fda854eaba29e51a70a3fb179" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38407 } }
I20260923 20:59:09.999495 22334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:09.999639 22334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eee576fda854eaba29e51a70a3fb179, State: Initialized, Role: FOLLOWER
I20260923 20:59:10.000399 22334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eee576fda854eaba29e51a70a3fb179" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38407 } }
I20260923 20:59:10.000677 22334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:10.000834 22334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:10.001017 22334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:10.003458 22334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eee576fda854eaba29e51a70a3fb179" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38407 } }
I20260923 20:59:10.004083 22334 leader_election.cc:302] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eee576fda854eaba29e51a70a3fb179; no voters:
I20260923 20:59:10.004643 22334 leader_election.cc:288] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:10.004825 22344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:10.005573 22344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [term 1 LEADER]: Becoming Leader. State: Replica: 6eee576fda854eaba29e51a70a3fb179, State: Running, Role: LEADER
I20260923 20:59:10.006203 22344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eee576fda854eaba29e51a70a3fb179" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38407 } }
I20260923 20:59:10.006875 22334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:10.010124 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eee576fda854eaba29e51a70a3fb179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eee576fda854eaba29e51a70a3fb179" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38407 } } }
I20260923 20:59:10.010550 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:10.011066 22346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eee576fda854eaba29e51a70a3fb179. Latest consensus state: current_term: 1 leader_uuid: "6eee576fda854eaba29e51a70a3fb179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eee576fda854eaba29e51a70a3fb179" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38407 } } }
I20260923 20:59:10.011384 22346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:10.012320 22349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:10.019541 22349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:10.030673 22349 catalog_manager.cc:1347] Generated new cluster ID: 0588875558594e3fa00e09895bf05dfb
I20260923 20:59:10.030817 22349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:10.039754 22349 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:10.041733 22349 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:10.055936 22349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eee576fda854eaba29e51a70a3fb179: Generated new TSK 0
I20260923 20:59:10.056908 22349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:10.071396 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22267
W20260923 20:59:10.073282 22329 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38109
W20260923 20:59:10.074236 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22267: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:59:10.074419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22267
W20260923 20:59:10.091737 22252 connection.cc:535] client connection to 127.0.97.62:38407 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38407): Connection reset by peer (error 104)
2026-09-23T20:59:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:10 dist-test-slave-plcg krb5kdc[22365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:10 dist-test-slave-plcg krb5kdc[22365](info): set up 2 sockets
Sep 23 20:59:10 dist-test-slave-plcg krb5kdc[22365](info): commencing operation
krb5kdc: starting...
W20260923 20:59:11.198460 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:11 dist-test-slave-plcg krb5kdc[22365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:11.474992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38851
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:11.822569 22378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:11.822893 22378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:11.823060 22378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:11.830679 22378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:11.830811 22378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:11.830899 22378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:11.831028 22378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:11.831106 22378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:11.841441 22378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:11.842907 22378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:11.844280 22378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:11.857503 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:11.857558 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:11.857577 22386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:11.858142 22378 server_base.cc:1034] running on GCE node
I20260923 20:59:11.859253 22378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:11.860906 22378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:11.862160 22378 hybrid_clock.cc:648] HybridClock initialized: now 1790197151862123 us; error 63 us; skew 500 ppm
I20260923 20:59:11.862569 22378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:11 dist-test-slave-plcg krb5kdc[22365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:11.866102 22378 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:11.867854 22378 webserver.cc:460] Webserver started at http://127.0.97.62:33781/ using document root <none> and password file <none>
I20260923 20:59:11.868634 22378 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:11.868801 22378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:11.869139 22378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:11.872524 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "168d9430822b4f85a77d40957c9920cf"
format_stamp: "Formatted at 2026-09-23 20:59:11 on dist-test-slave-plcg"
server_key: "f9b4d0bfee5833f3c177db6f35549ec2"
server_key_iv: "918a2d374bd2b4f08efc818a6d1ec071"
server_key_version: "encryptionkey@0"
I20260923 20:59:11.873405 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "168d9430822b4f85a77d40957c9920cf"
format_stamp: "Formatted at 2026-09-23 20:59:11 on dist-test-slave-plcg"
server_key: "f9b4d0bfee5833f3c177db6f35549ec2"
server_key_iv: "918a2d374bd2b4f08efc818a6d1ec071"
server_key_version: "encryptionkey@0"
I20260923 20:59:11.879459 22378 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 20:59:11.886574 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:11.888535 22378 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260923 20:59:11.888816 22378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "168d9430822b4f85a77d40957c9920cf"
format_stamp: "Formatted at 2026-09-23 20:59:11 on dist-test-slave-plcg"
server_key: "f9b4d0bfee5833f3c177db6f35549ec2"
server_key_iv: "918a2d374bd2b4f08efc818a6d1ec071"
server_key_version: "encryptionkey@0"
I20260923 20:59:11.889180 22378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:11.897459 22378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:11.902218 22378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:11.902560 22378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:11.923908 22378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38851
I20260923 20:59:11.924032 22444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38851 every 8 connection(s)
I20260923 20:59:11.925592 22378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:11.929796 22445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:11.932199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22378
I20260923 20:59:11.932397 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:11.932837 388 external_mini_cluster.cc:1428] Setting key d39efa95c47219d9eb5df1451f7eb4e8
I20260923 20:59:11.943178 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:11.934969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51633 (local address 127.0.97.62:38851)
0923 20:59:11.935914 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:11.935936 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:11.935998 (+ 62us) server_negotiation.cc:407] Connection header received
0923 20:59:11.938565 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:11.938600 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:11.939125 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:11.939639 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:11.940875 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:11.940885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:11.940905 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:11.940930 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:11.941008 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:11.941039 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:11.941318 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:11.941468 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 20:59:11.941709 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":400,"threads_started":1}
I20260923 20:59:11.945081 22445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf: Bootstrap starting.
I20260923 20:59:11.947827 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:11.949419 22445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:11.950939 22445 log.cc:824] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:11.954487 22445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf: No bootstrap required, opened a new log
I20260923 20:59:11.956280 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:11.953535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59241 (local address 127.0.97.62:38851)
0923 20:59:11.953764 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:11.953771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:11.954203 (+ 432us) server_negotiation.cc:407] Connection header received
0923 20:59:11.954394 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:11.954398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:11.954447 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:11.954538 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:11.955570 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:11.955578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:11.955582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:11.955598 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:11.955640 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:11.955655 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:11.955756 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:11.955900 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 20:59:11.955956 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:59:11.963392 22445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168d9430822b4f85a77d40957c9920cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38851 } }
I20260923 20:59:11.963893 22445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:11.964032 22445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168d9430822b4f85a77d40957c9920cf, State: Initialized, Role: FOLLOWER
I20260923 20:59:11.964797 22445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168d9430822b4f85a77d40957c9920cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38851 } }
I20260923 20:59:11.965044 22445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:11.965150 22445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:11.965282 22445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:11.967828 22445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168d9430822b4f85a77d40957c9920cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38851 } }
I20260923 20:59:11.968343 22445 leader_election.cc:302] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 168d9430822b4f85a77d40957c9920cf; no voters:
I20260923 20:59:11.968947 22445 leader_election.cc:288] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:11.969077 22455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:11.969907 22455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [term 1 LEADER]: Becoming Leader. State: Replica: 168d9430822b4f85a77d40957c9920cf, State: Running, Role: LEADER
I20260923 20:59:11.970907 22455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168d9430822b4f85a77d40957c9920cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38851 } }
I20260923 20:59:11.971557 22445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:11.974380 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 168d9430822b4f85a77d40957c9920cf. Latest consensus state: current_term: 1 leader_uuid: "168d9430822b4f85a77d40957c9920cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168d9430822b4f85a77d40957c9920cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38851 } } }
I20260923 20:59:11.974153 22456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "168d9430822b4f85a77d40957c9920cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168d9430822b4f85a77d40957c9920cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38851 } } }
I20260923 20:59:11.974876 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:11.974916 22456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:11.978967 22463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:11.984748 22463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:11.994681 22463 catalog_manager.cc:1347] Generated new cluster ID: 5232fa62051f47f28de685cd4b972b94
I20260923 20:59:11.994828 22463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:12.004624 22463 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:12.005165 22463 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:12.013937 22463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 168d9430822b4f85a77d40957c9920cf: Generated new TSK 0
I20260923 20:59:12.015141 22463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:12.024021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22378
W20260923 20:59:12.025446 22440 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51633
W20260923 20:59:12.026301 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22378: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:59:12.026476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22378
2026-09-23T20:59:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:12 dist-test-slave-plcg krb5kdc[22476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:12 dist-test-slave-plcg krb5kdc[22476](info): set up 2 sockets
Sep 23 20:59:12 dist-test-slave-plcg krb5kdc[22476](info): commencing operation
krb5kdc: starting...
W20260923 20:59:13.134449 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:13 dist-test-slave-plcg krb5kdc[22476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:13.406641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34747
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:13.756453 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:13.756771 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:13.756971 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:13.764539 22489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:13.764669 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:13.764777 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:13.764896 22489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:13.764981 22489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:13.775233 22489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:13.776731 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:13.778090 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:13.791033 22494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:13.791102 22497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:13.791085 22495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:13.792305 22489 server_base.cc:1034] running on GCE node
I20260923 20:59:13.793210 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:13.794831 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:13.796072 22489 hybrid_clock.cc:648] HybridClock initialized: now 1790197153796034 us; error 37 us; skew 500 ppm
I20260923 20:59:13.796535 22489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:13 dist-test-slave-plcg krb5kdc[22476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:13.799975 22489 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:13.801796 22489 webserver.cc:460] Webserver started at http://127.0.97.62:39503/ using document root <none> and password file <none>
I20260923 20:59:13.802636 22489 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:13.802803 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:13.803135 22489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:13.806444 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "8ecc196903464512a25a3841ad95797c"
format_stamp: "Formatted at 2026-09-23 20:59:13 on dist-test-slave-plcg"
server_key: "0cc5f9cc3ed0008e3cc8461d8cafb1bf"
server_key_iv: "a73ada058cb0325325a234401de307c5"
server_key_version: "encryptionkey@0"
I20260923 20:59:13.807361 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ecc196903464512a25a3841ad95797c"
format_stamp: "Formatted at 2026-09-23 20:59:13 on dist-test-slave-plcg"
server_key: "0cc5f9cc3ed0008e3cc8461d8cafb1bf"
server_key_iv: "a73ada058cb0325325a234401de307c5"
server_key_version: "encryptionkey@0"
I20260923 20:59:13.813575 22489 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 20:59:13.817682 22504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:13.819454 22489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:59:13.819710 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "8ecc196903464512a25a3841ad95797c"
format_stamp: "Formatted at 2026-09-23 20:59:13 on dist-test-slave-plcg"
server_key: "0cc5f9cc3ed0008e3cc8461d8cafb1bf"
server_key_iv: "a73ada058cb0325325a234401de307c5"
server_key_version: "encryptionkey@0"
I20260923 20:59:13.819977 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:13.840911 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:13.845424 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:13.845759 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:13.865164 22489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34747
I20260923 20:59:13.865185 22555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34747 every 8 connection(s)
I20260923 20:59:13.866691 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:13.870719 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:13.874182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22489
I20260923 20:59:13.874384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:13.875028 388 external_mini_cluster.cc:1428] Setting key 26efd3e614fa2aa416e26c37a6859b95
I20260923 20:59:13.884735 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c: Bootstrap starting.
I20260923 20:59:13.890030 22556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:13.891564 22556 log.cc:824] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:59:13 dist-test-slave-plcg krb5kdc[22476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:59:13.895100 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c: No bootstrap required, opened a new log
I20260923 20:59:13.902666 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:13.877256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57023 (local address 127.0.97.62:34747)
0923 20:59:13.878187 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:13.878200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:13.878246 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:59:13.880019 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:13.880046 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:13.880522 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:13.881019 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:13.882737 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:13.885466 (+ 2729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:13.887940 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:13.889128 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:13.889353 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:13.894270 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:13.894279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:13.894296 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:13.894338 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:13.897715 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:13.898644 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:13.898652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:13.898660 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:13.898896 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:13.899339 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:13.899350 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:13.899352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:13.899992 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:13.900357 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:13.900874 (+ 517us) server_negotiation.cc:299] Negotiation successful
0923 20:59:13.901425 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":426,"threads_started":1}
I20260923 20:59:13.904130 22556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecc196903464512a25a3841ad95797c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34747 } }
I20260923 20:59:13.904687 22556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:13.904814 22556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecc196903464512a25a3841ad95797c, State: Initialized, Role: FOLLOWER
I20260923 20:59:13.905524 22556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecc196903464512a25a3841ad95797c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34747 } }
I20260923 20:59:13.905885 22556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:13.906109 22556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:13.906333 22556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:13.907143 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:13.910003 22556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecc196903464512a25a3841ad95797c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34747 } }
I20260923 20:59:13.910913 22556 leader_election.cc:302] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ecc196903464512a25a3841ad95797c; no voters:
I20260923 20:59:13.911777 22556 leader_election.cc:288] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:13.911861 22565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:13.913448 22565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [term 1 LEADER]: Becoming Leader. State: Replica: 8ecc196903464512a25a3841ad95797c, State: Running, Role: LEADER
I20260923 20:59:13.914202 22565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecc196903464512a25a3841ad95797c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34747 } }
I20260923 20:59:13.914696 22556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:13.917337 22567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ecc196903464512a25a3841ad95797c. Latest consensus state: current_term: 1 leader_uuid: "8ecc196903464512a25a3841ad95797c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecc196903464512a25a3841ad95797c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34747 } } }
I20260923 20:59:13.917718 22567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:13.918063 22566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ecc196903464512a25a3841ad95797c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ecc196903464512a25a3841ad95797c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34747 } } }
I20260923 20:59:13.918424 22566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:13.933415 22572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:13.939087 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:13.913128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35239 (local address 127.0.97.62:34747)
0923 20:59:13.913361 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:13.913367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:13.914253 (+ 886us) server_negotiation.cc:407] Connection header received
0923 20:59:13.914457 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:13.914464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:13.914558 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:13.914699 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:13.919248 (+ 4549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:13.920801 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:13.923617 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:13.924697 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:13.924847 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:13.929380 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:13.929392 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:13.929401 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:13.929478 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:13.934655 (+ 5177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:13.935727 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:13.935734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:13.935737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:13.935929 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:13.936475 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:13.936480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:13.936483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:13.936951 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:13.937236 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:13.938470 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0923 20:59:13.938715 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:59:13.939769 22572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:13.950980 22572 catalog_manager.cc:1347] Generated new cluster ID: 7f90cb1db20847b4a434bab669cb448c
I20260923 20:59:13.951124 22572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:13.973074 22572 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:13.974879 22572 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:13.989719 22572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ecc196903464512a25a3841ad95797c: Generated new TSK 0
I20260923 20:59:13.990789 22572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:14.001282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22489
I20260923 20:59:14.002574 22551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:14.096860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22489
2026-09-23T20:59:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:14 dist-test-slave-plcg krb5kdc[22588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:14 dist-test-slave-plcg krb5kdc[22588](info): set up 2 sockets
Sep 23 20:59:14 dist-test-slave-plcg krb5kdc[22588](info): commencing operation
krb5kdc: starting...
W20260923 20:59:15.206853 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:15 dist-test-slave-plcg krb5kdc[22588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:15.494954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40421
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:15.844122 22601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:15.844552 22601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:15.844744 22601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:15.852572 22601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:15.852782 22601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:15.852907 22601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:15.853014 22601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:15.853101 22601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:15.863660 22601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:15.865141 22601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:15.866580 22601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:15.880645 22606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:15.880669 22607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:15.880620 22609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:15.881381 22601 server_base.cc:1034] running on GCE node
I20260923 20:59:15.882334 22601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:15.885059 22601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:15.886371 22601 hybrid_clock.cc:648] HybridClock initialized: now 1790197155886277 us; error 67 us; skew 500 ppm
I20260923 20:59:15.886950 22601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:15 dist-test-slave-plcg krb5kdc[22588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:15.896649 22601 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:15.898557 22601 webserver.cc:460] Webserver started at http://127.0.97.62:37953/ using document root <none> and password file <none>
I20260923 20:59:15.899471 22601 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:15.899669 22601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:15.900120 22601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:15.903795 22601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "91285966a1bb4b099fdc79d7933968dd"
format_stamp: "Formatted at 2026-09-23 20:59:15 on dist-test-slave-plcg"
server_key: "f491bf0ba3b6066bd4bcda5999ae142b"
server_key_iv: "a004bacf73ffa4e7e0d2d0633fe56ff8"
server_key_version: "encryptionkey@0"
I20260923 20:59:15.904644 22601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "91285966a1bb4b099fdc79d7933968dd"
format_stamp: "Formatted at 2026-09-23 20:59:15 on dist-test-slave-plcg"
server_key: "f491bf0ba3b6066bd4bcda5999ae142b"
server_key_iv: "a004bacf73ffa4e7e0d2d0633fe56ff8"
server_key_version: "encryptionkey@0"
I20260923 20:59:15.910665 22601 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:59:15.915076 22616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:15.917452 22601 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:59:15.917761 22601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "91285966a1bb4b099fdc79d7933968dd"
format_stamp: "Formatted at 2026-09-23 20:59:15 on dist-test-slave-plcg"
server_key: "f491bf0ba3b6066bd4bcda5999ae142b"
server_key_iv: "a004bacf73ffa4e7e0d2d0633fe56ff8"
server_key_version: "encryptionkey@0"
I20260923 20:59:15.918071 22601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:15.934825 22601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:15.939666 22601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:15.940044 22601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:15.960206 22601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40421
I20260923 20:59:15.960233 22667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40421 every 8 connection(s)
I20260923 20:59:15.961781 22601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:15.963290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22601
I20260923 20:59:15.963472 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:15.964031 388 external_mini_cluster.cc:1428] Setting key debb9521899c2c41fe96f073b3843e01
I20260923 20:59:15.966913 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:15.978941 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:15.966082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37805 (local address 127.0.97.62:40421)
0923 20:59:15.966929 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:15.966942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:15.967016 (+ 74us) server_negotiation.cc:407] Connection header received
0923 20:59:15.968747 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:15.968772 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:15.969205 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:15.969677 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:15.970372 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:15.971728 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:15.973163 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:15.974058 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:15.974274 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:15.976102 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:15.976114 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:15.976132 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:15.976815 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:15.976909 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:15.976945 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:15.977164 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:15.977457 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 20:59:15.977780 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":369,"threads_started":1}
I20260923 20:59:15.980182 22668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd: Bootstrap starting.
I20260923 20:59:15.983206 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:15.983899 22668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:15.985026 22668 log.cc:824] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:15.989801 22668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd: No bootstrap required, opened a new log
I20260923 20:59:15.996326 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:15.988907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51873 (local address 127.0.97.62:40421)
0923 20:59:15.989116 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:15.989125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:15.989659 (+ 534us) server_negotiation.cc:407] Connection header received
0923 20:59:15.990086 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:15.990091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:15.990167 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:15.990296 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:15.990912 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:15.991748 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:15.993316 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:15.993957 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:15.994150 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:15.995467 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:15.995472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:15.995476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:15.995498 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:15.995547 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:15.995564 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:15.995692 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:15.995858 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 20:59:15.995968 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:59:16.000825 22668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91285966a1bb4b099fdc79d7933968dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40421 } }
I20260923 20:59:16.001302 22668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:16.001469 22668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91285966a1bb4b099fdc79d7933968dd, State: Initialized, Role: FOLLOWER
I20260923 20:59:16.002213 22668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91285966a1bb4b099fdc79d7933968dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40421 } }
I20260923 20:59:16.002444 22668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:16.002614 22668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:16.002810 22668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:16.005178 22668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91285966a1bb4b099fdc79d7933968dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40421 } }
I20260923 20:59:16.005795 22668 leader_election.cc:302] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91285966a1bb4b099fdc79d7933968dd; no voters:
I20260923 20:59:16.006623 22668 leader_election.cc:288] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:16.006738 22678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:16.007452 22678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [term 1 LEADER]: Becoming Leader. State: Replica: 91285966a1bb4b099fdc79d7933968dd, State: Running, Role: LEADER
I20260923 20:59:16.008085 22678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91285966a1bb4b099fdc79d7933968dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40421 } }
I20260923 20:59:16.009301 22668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:16.011333 22680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91285966a1bb4b099fdc79d7933968dd. Latest consensus state: current_term: 1 leader_uuid: "91285966a1bb4b099fdc79d7933968dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91285966a1bb4b099fdc79d7933968dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40421 } } }
I20260923 20:59:16.011849 22680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:16.014400 22683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:16.014400 22679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91285966a1bb4b099fdc79d7933968dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91285966a1bb4b099fdc79d7933968dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40421 } } }
I20260923 20:59:16.015064 22679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:16.019714 22683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:16.030212 22683 catalog_manager.cc:1347] Generated new cluster ID: 63a757928dbf4741bccaff0ec7ea04da
I20260923 20:59:16.030391 22683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:16.049479 22683 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:16.051108 22683 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:16.061683 22683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91285966a1bb4b099fdc79d7933968dd: Generated new TSK 0
I20260923 20:59:16.062711 22683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:16.074920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22601
W20260923 20:59:16.076094 22663 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37805
W20260923 20:59:16.076855 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22601: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:59:16.077023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22601
2026-09-23T20:59:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:16 dist-test-slave-plcg krb5kdc[22699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:16 dist-test-slave-plcg krb5kdc[22699](info): set up 2 sockets
Sep 23 20:59:16 dist-test-slave-plcg krb5kdc[22699](info): commencing operation
krb5kdc: starting...
W20260923 20:59:17.190450 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:17 dist-test-slave-plcg krb5kdc[22699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:17.473098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39621
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:17.826553 22712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:17.826894 22712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:17.827080 22712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:17.835088 22712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:17.835271 22712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:17.835386 22712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:17.835491 22712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:17.835573 22712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:17.846292 22712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:17.847694 22712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:17.849166 22712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:17.862380 22717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:17.862438 22718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:17.862474 22720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:17.864048 22712 server_base.cc:1034] running on GCE node
I20260923 20:59:17.864974 22712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:17.866649 22712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:17.867913 22712 hybrid_clock.cc:648] HybridClock initialized: now 1790197157867864 us; error 41 us; skew 500 ppm
I20260923 20:59:17.868320 22712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:17 dist-test-slave-plcg krb5kdc[22699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:17.877667 22712 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:17.879602 22712 webserver.cc:460] Webserver started at http://127.0.97.62:40597/ using document root <none> and password file <none>
I20260923 20:59:17.880403 22712 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:17.880555 22712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:17.880905 22712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:17.885088 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "a9414b6e62f1445eb9b11cfa8fdae25f"
format_stamp: "Formatted at 2026-09-23 20:59:17 on dist-test-slave-plcg"
server_key: "280414de57de407b6d28d80a372f6d4a"
server_key_iv: "367efca9be5b349a8ada50e7d5ed12b4"
server_key_version: "encryptionkey@0"
I20260923 20:59:17.886335 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "a9414b6e62f1445eb9b11cfa8fdae25f"
format_stamp: "Formatted at 2026-09-23 20:59:17 on dist-test-slave-plcg"
server_key: "280414de57de407b6d28d80a372f6d4a"
server_key_iv: "367efca9be5b349a8ada50e7d5ed12b4"
server_key_version: "encryptionkey@0"
I20260923 20:59:17.893978 22712 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260923 20:59:17.898052 22727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:17.899801 22712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:59:17.900053 22712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "a9414b6e62f1445eb9b11cfa8fdae25f"
format_stamp: "Formatted at 2026-09-23 20:59:17 on dist-test-slave-plcg"
server_key: "280414de57de407b6d28d80a372f6d4a"
server_key_iv: "367efca9be5b349a8ada50e7d5ed12b4"
server_key_version: "encryptionkey@0"
I20260923 20:59:17.900298 22712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:17.906656 22712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:17.911306 22712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:17.911643 22712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:17.931754 22712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39621
I20260923 20:59:17.931851 22778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39621 every 8 connection(s)
I20260923 20:59:17.933279 22712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:17.937326 22779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:17.940580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22712
I20260923 20:59:17.940766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:17.941303 388 external_mini_cluster.cc:1428] Setting key 022e3ef47df46a514702f2201d054760
I20260923 20:59:17.951970 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:17.943924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37833 (local address 127.0.97.62:39621)
0923 20:59:17.945107 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:17.945130 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:17.945642 (+ 512us) server_negotiation.cc:407] Connection header received
0923 20:59:17.947653 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:17.947696 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:17.948271 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:17.948794 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:17.949901 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:17.949909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:17.949928 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:17.949952 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:17.950028 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:17.950055 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:17.950308 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:17.950442 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 20:59:17.950639 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":527,"threads_started":1}
I20260923 20:59:17.952589 22779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f: Bootstrap starting.
I20260923 20:59:17.956295 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:17.956375 22779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:17.957544 22779 log.cc:824] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:17.961164 22779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f: No bootstrap required, opened a new log
I20260923 20:59:17.964864 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:17.961779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57095 (local address 127.0.97.62:39621)
0923 20:59:17.961992 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:17.961999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:17.962626 (+ 627us) server_negotiation.cc:407] Connection header received
0923 20:59:17.962811 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:17.962815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:17.962872 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:17.962979 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:17.964013 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:17.964018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:17.964021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:59:17.964040 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:17.964087 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:59:17.964112 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:17.964221 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:17.964404 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 20:59:17.964519 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:59:17.970346 22779 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260923 20:59:17.970849 22779 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:59:17.970997 22779 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9414b6e62f1445eb9b11cfa8fdae25f, State: Initialized, Role: FOLLOWER
I20260923 20:59:17.971691 22779 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260923 20:59:17.971915 22779 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:59:17.972029 22779 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:59:17.972160 22779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:17.974525 22779 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260923 20:59:17.975180 22779 leader_election.cc:302] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9414b6e62f1445eb9b11cfa8fdae25f; no voters:
I20260923 20:59:17.976217 22779 leader_election.cc:288] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:59:17.976358 22789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:17.977259 22789 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [term 1 LEADER]: Becoming Leader. State: Replica: a9414b6e62f1445eb9b11cfa8fdae25f, State: Running, Role: LEADER
I20260923 20:59:17.977991 22789 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260923 20:59:17.978859 22779 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:17.983142 22791 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9414b6e62f1445eb9b11cfa8fdae25f. Latest consensus state: current_term: 1 leader_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } } }
I20260923 20:59:17.983700 22791 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:17.986377 22790 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9414b6e62f1445eb9b11cfa8fdae25f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } } }
I20260923 20:59:17.986833 22790 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f [sys.catalog]: This master's current role is: LEADER
W20260923 20:59:18.002420 22805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:59:18.002624 22805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:59:18.002830 22795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:18.005601 22795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:18.016522 22795 catalog_manager.cc:1347] Generated new cluster ID: 8ca0e658e8e24fd7b1e5993250d638a1
I20260923 20:59:18.016702 22795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:18.039525 22795 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:18.040124 22795 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:18.050912 22795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9414b6e62f1445eb9b11cfa8fdae25f: Generated new TSK 0
I20260923 20:59:18.052094 22795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:18.070812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22712
W20260923 20:59:18.072234 22774 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37833
W20260923 20:59:18.073056 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22712: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:59:18.073226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22712
2026-09-23T20:59:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1995 ms)
[----------] 18 tests from AuthTokenIssuingTest (36059 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:18 dist-test-slave-plcg krb5kdc[22810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:18 dist-test-slave-plcg krb5kdc[22810](info): set up 2 sockets
Sep 23 20:59:18 dist-test-slave-plcg krb5kdc[22810](info): commencing operation
krb5kdc: starting...
W20260923 20:59:19.190505 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:19 dist-test-slave-plcg krb5kdc[22810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:19.473521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38537
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38537,127.0.97.61:44263,127.0.97.60:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:19.822316 22823 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:59:19.822674 22823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:19.822750 22823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:19.822836 22823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:19.830384 22823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:19.830499 22823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:19.830564 22823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:19.830608 22823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:19.830673 22823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:19.840885 22823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38537,127.0.97.61:44263,127.0.97.60:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:19.842260 22823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:19.843714 22823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:19.856899 22831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:19.857021 22828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:19.856994 22829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:19.857895 22823 server_base.cc:1034] running on GCE node
I20260923 20:59:19.858816 22823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:19.860476 22823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:19.861785 22823 hybrid_clock.cc:648] HybridClock initialized: now 1790197159861705 us; error 59 us; skew 500 ppm
I20260923 20:59:19.862253 22823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:19 dist-test-slave-plcg krb5kdc[22810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:19.865993 22823 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:19.867759 22823 webserver.cc:460] Webserver started at http://127.0.97.62:39469/ using document root <none> and password file <none>
I20260923 20:59:19.868633 22823 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:19.868806 22823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:19.869206 22823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:19.872691 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "c7493aa985104d3498e3d0a9efcae354"
format_stamp: "Formatted at 2026-09-23 20:59:19 on dist-test-slave-plcg"
server_key: "e67181d1aedb089c45cf97b5bdfda874"
server_key_iv: "c980950bbbe90fb7cc464ba526f063e2"
server_key_version: "encryptionkey@0"
I20260923 20:59:19.873649 22823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "c7493aa985104d3498e3d0a9efcae354"
format_stamp: "Formatted at 2026-09-23 20:59:19 on dist-test-slave-plcg"
server_key: "e67181d1aedb089c45cf97b5bdfda874"
server_key_iv: "c980950bbbe90fb7cc464ba526f063e2"
server_key_version: "encryptionkey@0"
I20260923 20:59:19.879673 22823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:59:19.884123 22838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:19.886456 22823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 20:59:19.886696 22823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "c7493aa985104d3498e3d0a9efcae354"
format_stamp: "Formatted at 2026-09-23 20:59:19 on dist-test-slave-plcg"
server_key: "e67181d1aedb089c45cf97b5bdfda874"
server_key_iv: "c980950bbbe90fb7cc464ba526f063e2"
server_key_version: "encryptionkey@0"
I20260923 20:59:19.886965 22823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:19.917384 22823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:19.922065 22823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:19.922372 22823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:19.942322 22823 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38537
I20260923 20:59:19.942340 22889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38537 every 8 connection(s)
I20260923 20:59:19.943959 22823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:19.948144 22890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:19.950892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22823
I20260923 20:59:19.951138 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:19.951778 388 external_mini_cluster.cc:1428] Setting key cc5babfb84f122b66fe5bd9f97d7825e
I20260923 20:59:19.955453 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:19 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260923 20:59:19.984604 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:19.959200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38537 (local address 127.0.0.1:37880)
0923 20:59:19.960522 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:19.960593 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:19.963757 (+ 3164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:19.964953 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:19.964963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:19.965891 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:19.966472 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:19.966505 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:19.968290 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:19.968298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:19.970453 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:19.970478 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:19.971674 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:19.971684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:19.971926 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:19.972739 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:19.972777 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:19.975916 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:19.979969 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:19.979983 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:19.979997 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:19.980428 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:19.980891 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:19.980899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:19.980903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:19.981077 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:19.982025 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:19.982039 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:19.982294 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 20:59:19.982561 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 20:59:19.983216 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":172,"thread_start_us":639,"threads_started":1}
I20260923 20:59:19.984642 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:19.959200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.0.97.62:38537)
0923 20:59:19.961605 (+ 2405us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:19.961612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:19.961658 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:59:19.964304 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:19.964311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:19.964637 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:19.964824 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:19.966767 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:19.968078 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:19.970682 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:19.971349 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:19.971526 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:19.976238 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:19.976248 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:19.976262 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:19.976299 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:19.979771 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:19.980607 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:19.980615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:19.980617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:19.980768 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:19.981199 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:19.981203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:19.981206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:19.981864 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:19.982114 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:19.982703 (+ 589us) server_negotiation.cc:299] Negotiation successful
0923 20:59:19.983076 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2194,"spinlock_wait_cycles":18304,"thread_start_us":1914,"threads_started":1}
I20260923 20:59:19.987802 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:19.989210 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:19.988731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46940)
0923 20:59:19.988924 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:19.989065 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 20:59:19.989534 22840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
I20260923 20:59:19.991362 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:19.990957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46950)
0923 20:59:19.991121 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:19.991217 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:59:19.991652 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44263: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 20:59:20.017233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44263
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38537,127.0.97.61:44263,127.0.97.60:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:59:20.041081 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } attempt: 1
I20260923 20:59:20.042366 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.041917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46956)
0923 20:59:20.042113 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.042208 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 20:59:20.043500 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.043053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46964)
0923 20:59:20.043242 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.043354 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:59:20.043726 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44263: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
I20260923 20:59:20.112052 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } attempt: 2
I20260923 20:59:20.113305 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.112796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46966)
0923 20:59:20.113074 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.113160 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:59:20.114473 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.114067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46970)
0923 20:59:20.114225 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.114323 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:59:20.114678 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44263: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
I20260923 20:59:20.205986 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } attempt: 3
I20260923 20:59:20.207286 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.206717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46978)
0923 20:59:20.206978 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.207097 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:59:20.208473 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.208019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46982)
0923 20:59:20.208197 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.208291 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 20:59:20.208703 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44263: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
I20260923 20:59:20.351996 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } attempt: 4
I20260923 20:59:20.353402 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.352751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46986)
0923 20:59:20.353124 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.353237 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260923 20:59:20.354679 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.354182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46992)
0923 20:59:20.354357 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.354455 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:59:20.354983 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44263: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44263: connect: Connection refused (error 111)
W20260923 20:59:20.372170 22897 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:59:20.372601 22897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:20.372731 22897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:20.372874 22897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:20.380450 22897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:20.380616 22897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:20.380728 22897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:20.380820 22897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:20.380913 22897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:20.391528 22897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38537,127.0.97.61:44263,127.0.97.60:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:20.392944 22897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:20.394446 22897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:20.407390 22902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:20.407485 22903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:20.407433 22905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:20.414120 22897 server_base.cc:1034] running on GCE node
I20260923 20:59:20.415071 22897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:20.416636 22897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:20.417964 22897 hybrid_clock.cc:648] HybridClock initialized: now 1790197160417883 us; error 44 us; skew 500 ppm
I20260923 20:59:20.418340 22897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:20 dist-test-slave-plcg krb5kdc[22810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:20.421736 22897 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 20:59:20.423468 22897 webserver.cc:460] Webserver started at http://127.0.97.61:43593/ using document root <none> and password file <none>
I20260923 20:59:20.424280 22897 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:20.424439 22897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:20.424863 22897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:20.428120 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data/instance:
uuid: "0da6fca763c847419376e9f7c680f2ae"
format_stamp: "Formatted at 2026-09-23 20:59:20 on dist-test-slave-plcg"
server_key: "e7dcdce1697867dfdbc93b279e2677c8"
server_key_iv: "cb96ffd3903f6f189614f225d4fe832a"
server_key_version: "encryptionkey@0"
I20260923 20:59:20.429013 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/wal/instance:
uuid: "0da6fca763c847419376e9f7c680f2ae"
format_stamp: "Formatted at 2026-09-23 20:59:20 on dist-test-slave-plcg"
server_key: "e7dcdce1697867dfdbc93b279e2677c8"
server_key_iv: "cb96ffd3903f6f189614f225d4fe832a"
server_key_version: "encryptionkey@0"
I20260923 20:59:20.434990 22897 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:59:20.438978 22912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:20.440650 22897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:59:20.440881 22897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/wal
uuid: "0da6fca763c847419376e9f7c680f2ae"
format_stamp: "Formatted at 2026-09-23 20:59:20 on dist-test-slave-plcg"
server_key: "e7dcdce1697867dfdbc93b279e2677c8"
server_key_iv: "cb96ffd3903f6f189614f225d4fe832a"
server_key_version: "encryptionkey@0"
I20260923 20:59:20.441150 22897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:20.474723 22897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:20.479321 22897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:20.479641 22897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:20.499666 22897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44263
I20260923 20:59:20.499737 22963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44263 every 8 connection(s)
I20260923 20:59:20.501187 22897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
I20260923 20:59:20.505328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22897
I20260923 20:59:20.505366 22964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:20.505533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-1/wal/instance
I20260923 20:59:20.506089 388 external_mini_cluster.cc:1428] Setting key cdf6f6cb43524df5f1e3110db40c5de2
I20260923 20:59:20.516073 22964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:20 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 20:59:20.539619 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.519436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.0.97.62:38537)
0923 20:59:20.519949 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:20.519954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:20.521119 (+ 1165us) server_negotiation.cc:407] Connection header received
0923 20:59:20.523260 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:20.523266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:20.523369 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:20.523502 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:20.525257 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.526064 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.527770 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.528200 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.528352 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:20.532276 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:20.532282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:20.532285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:20.532338 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:20.535453 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:20.536638 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:20.536643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:20.536645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:20.536815 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:20.537436 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:20.537440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:20.537442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:20.537779 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:20.538001 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:20.539073 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0923 20:59:20.539315 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":286,"threads_started":1}
I20260923 20:59:20.541245 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.520075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38537 (local address 127.0.0.1:37886)
0923 20:59:20.520936 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.520982 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:20.522936 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:20.523787 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:20.523797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:20.524384 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:20.525006 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:20.525024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.526252 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.526257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:20.527588 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:20.527602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.528466 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.528473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:20.528727 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:20.529558 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:20.529584 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:20.531988 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:20.535659 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:20.535670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:20.535691 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:20.536448 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:20.536973 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:20.536981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:20.536987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:20.537274 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:20.538001 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:20.538014 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:20.538614 (+ 600us) client_negotiation.cc:769] Sending connection context
0923 20:59:20.538948 (+ 334us) client_negotiation.cc:240] Negotiation successful
0923 20:59:20.539573 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":346,"threads_started":1}
I20260923 20:59:20.543584 22964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:20 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:20.561085 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.544616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:46998)
0923 20:59:20.544853 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.544879 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:20.544983 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:20.546117 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:20.546125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:20.546681 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:20.546845 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:20.546854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.548378 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.548386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:20.550061 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:20.550081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.551027 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.551039 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:20.551274 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:20.551991 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:20.552021 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:20.554757 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:20.558317 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:20.558322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:20.558326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:20.558738 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:20.559177 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:20.559181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:20.559185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:20.559331 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:20.559959 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:20.559964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:20.560132 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 20:59:20.560290 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 20:59:20.560653 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:59:20.561180 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.544762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.0.97.61:44263)
0923 20:59:20.545485 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:20.545492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:20.545517 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:59:20.545568 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:20.545574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:20.545895 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:20.546100 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:20.547043 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.548196 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.550242 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.550844 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.550966 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:20.555016 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:20.555032 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:20.555048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:20.555080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:20.558140 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:20.558906 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:20.558913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:20.558915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:20.559045 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:20.559448 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:20.559452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:20.559455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:20.559825 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:20.560068 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:20.560429 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 20:59:20.560786 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":566,"thread_start_us":421,"threads_started":1}
I20260923 20:59:20.563973 22964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:20.565327 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.564824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36426)
0923 20:59:20.565016 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.565167 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 20:59:20.565642 22916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.567440 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.567030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36438)
0923 20:59:20.567231 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.567312 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 20:59:20.567682 22964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 20:59:20.574986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45171
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38537,127.0.97.61:44263,127.0.97.60:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:59:20.616997 22964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 1
I20260923 20:59:20.618187 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.617681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36444)
0923 20:59:20.617913 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.618012 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:59:20.619292 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.618916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36452)
0923 20:59:20.619089 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.619172 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:59:20.619529 22964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.654268 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } attempt: 5
Sep 23 20:59:20 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:20.669119 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.655116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.0.97.61:44263)
0923 20:59:20.655349 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:20.655355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:20.655381 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:59:20.655465 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:20.655471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:20.655554 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:20.655684 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:20.656454 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.657553 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.658888 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.659738 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.659932 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:20.663138 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:20.663145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:20.663150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:20.663193 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:20.666388 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:20.667135 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:20.667141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:20.667144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:20.667277 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:20.667666 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:20.667672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:20.667675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:20.667972 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:20.668137 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:20.668538 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 20:59:20.668776 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:59:20.669143 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.655012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:47006)
0923 20:59:20.655218 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.655239 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:20.655351 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:20.655715 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:20.655718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:20.656108 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:20.656282 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:20.656291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.657742 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.657746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:20.658704 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:20.658717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:20.659961 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:20.659966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:20.660155 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:20.660830 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:20.660847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:20.662903 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:20.666572 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:20.666578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:20.666582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:20.666994 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:20.667400 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:20.667405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:20.667408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:20.667550 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:20.668099 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:20.668104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:20.668263 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 20:59:20.668422 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 20:59:20.668773 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 20:59:20.670409 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:20.671653 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.671203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36454)
0923 20:59:20.671409 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.671492 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 20:59:20.672643 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.672300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36460)
0923 20:59:20.672451 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.672541 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:59:20.672863 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.687753 22964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 2
I20260923 20:59:20.688884 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.688429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36470)
0923 20:59:20.688628 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.688727 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:59:20.689935 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.689529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36476)
0923 20:59:20.689674 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.689747 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:59:20.690142 22964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.724114 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 1
I20260923 20:59:20.725442 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.724959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36488)
0923 20:59:20.725173 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.725265 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:59:20.726564 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.726220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36500)
0923 20:59:20.726376 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.726454 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:59:20.726958 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.781384 22964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 3
I20260923 20:59:20.782605 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.782127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36506)
0923 20:59:20.782357 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.782440 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 20:59:20.783653 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.783236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36522)
0923 20:59:20.783469 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.783542 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260923 20:59:20.783847 22964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.795192 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 2
I20260923 20:59:20.796346 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.795845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36528)
0923 20:59:20.796088 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.796177 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:59:20.797425 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.797062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36530)
0923 20:59:20.797241 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.797311 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:59:20.797639 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.903882 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 3
I20260923 20:59:20.905191 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.904690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36532)
0923 20:59:20.904930 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.905027 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 20:59:20.906414 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.906020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36544)
0923 20:59:20.906236 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.906309 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260923 20:59:20.906621 22890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
I20260923 20:59:20.927053 22964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 4
I20260923 20:59:20.928314 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.927705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36558)
0923 20:59:20.927992 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.928105 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:59:20.929447 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:20.929048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36570)
0923 20:59:20.929213 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:20.929304 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:59:20.929729 22964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45171: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45171: connect: Connection refused (error 111)
W20260923 20:59:20.932906 22972 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:59:20.933300 22972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:20.933427 22972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:20.933564 22972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:20.941500 22972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:20.941660 22972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:20.941774 22972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:20.941881 22972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:20.941984 22972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:20.953011 22972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38537,127.0.97.61:44263,127.0.97.60:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:20.954550 22972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:20.955947 22972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:20.968693 22977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:20.968670 22978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:20.968631 22980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:20.969658 22972 server_base.cc:1034] running on GCE node
I20260923 20:59:20.970547 22972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:20.972132 22972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:20.973528 22972 hybrid_clock.cc:648] HybridClock initialized: now 1790197160973440 us; error 64 us; skew 500 ppm
I20260923 20:59:20.974020 22972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:20 dist-test-slave-plcg krb5kdc[22810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:20.988966 22972 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 20:59:20.990854 22972 webserver.cc:460] Webserver started at http://127.0.97.60:39641/ using document root <none> and password file <none>
I20260923 20:59:20.991750 22972 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:20.991925 22972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:20.992223 22972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:20.995625 22972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data/instance:
uuid: "612a0fa714c547019ef09237cf3ed61a"
format_stamp: "Formatted at 2026-09-23 20:59:20 on dist-test-slave-plcg"
server_key: "76130fadfc7805abf69bbd6432915b45"
server_key_iv: "43dc1bd8c7cf341bb6ac006475e66599"
server_key_version: "encryptionkey@0"
I20260923 20:59:20.996461 22972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/wal/instance:
uuid: "612a0fa714c547019ef09237cf3ed61a"
format_stamp: "Formatted at 2026-09-23 20:59:20 on dist-test-slave-plcg"
server_key: "76130fadfc7805abf69bbd6432915b45"
server_key_iv: "43dc1bd8c7cf341bb6ac006475e66599"
server_key_version: "encryptionkey@0"
I20260923 20:59:21.002287 22972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:59:21.006127 22987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:21.007671 22972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:59:21.007902 22972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/wal
uuid: "612a0fa714c547019ef09237cf3ed61a"
format_stamp: "Formatted at 2026-09-23 20:59:20 on dist-test-slave-plcg"
server_key: "76130fadfc7805abf69bbd6432915b45"
server_key_iv: "43dc1bd8c7cf341bb6ac006475e66599"
server_key_version: "encryptionkey@0"
I20260923 20:59:21.008175 22972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:21.031173 22972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:21.035557 22972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:21.035877 22972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:21.057401 22972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45171
I20260923 20:59:21.057437 23038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45171 every 8 connection(s)
I20260923 20:59:21.058887 22972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
I20260923 20:59:21.063160 23039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:21.063215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 22972
I20260923 20:59:21.063427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790196790989944-388-0/minicluster-data/master-2/wal/instance
I20260923 20:59:21.063932 388 external_mini_cluster.cc:1428] Setting key 5c392587d6522f81dcb1974e18bb716f
I20260923 20:59:21.074014 23039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:59:21.083848 22890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 4
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:59:21.090448 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.065972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.0.97.60:45171)
0923 20:59:21.066875 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.066890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.066943 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:59:21.068589 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.068622 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.069179 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.069677 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.071121 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.073099 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.074551 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.075669 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.075856 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.081194 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.081203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.081215 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.081258 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.085144 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.086127 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.086136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.086146 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.086379 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.086869 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.086875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.086878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.087552 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.087946 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.088362 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.088857 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":372,"threads_started":1}
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:59:21.096726 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.077177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.0.97.62:38537)
0923 20:59:21.077765 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.077770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.078220 (+ 450us) server_negotiation.cc:407] Connection header received
0923 20:59:21.078440 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.078450 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.078517 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.078640 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.079999 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.080828 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.082617 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.083333 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.083494 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.088299 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.088305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.088308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.088341 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.092283 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.093293 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.093299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.093302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.093467 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.094519 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.094527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.094530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.094936 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.095115 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.095957 (+ 842us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.096227 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":268,"threads_started":1}
I20260923 20:59:21.097208 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.077209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38537 (local address 127.0.0.1:37890)
0923 20:59:21.078076 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:21.078119 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:21.078307 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:21.078713 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:21.078720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:21.079438 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:21.079799 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.079816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.081013 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.081021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.082431 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.082446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.083545 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.083550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.083688 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.084462 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:21.084488 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:21.088077 (+ 3589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:21.092482 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.092493 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.092500 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.093167 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.093617 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.093621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.093624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.093892 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.095111 (+ 1219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:21.095129 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:21.095516 (+ 387us) client_negotiation.cc:769] Sending connection context
0923 20:59:21.095797 (+ 281us) client_negotiation.cc:240] Negotiation successful
0923 20:59:21.096322 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":471,"threads_started":1}
I20260923 20:59:21.102640 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.084615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.0.97.60:45171)
0923 20:59:21.085295 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.085300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.085322 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:59:21.085372 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.085376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.085443 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.085574 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.086548 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.087939 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.089796 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.090505 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.091283 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.094979 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.094990 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.094997 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.095041 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.099379 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.100155 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.100161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.100165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.100303 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.100724 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.100731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.100734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.101136 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.101298 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.101945 (+ 647us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.102204 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":303,"threads_started":1}
I20260923 20:59:21.103062 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.084552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36594)
0923 20:59:21.084757 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:21.084780 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:21.084914 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:21.085601 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:21.085607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:21.086149 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:21.086366 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.086378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.088071 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.088077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.089599 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.089620 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.090699 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.090707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.090946 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.091601 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:21.091648 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:21.094709 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:21.099528 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.099534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.099538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.100016 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.100439 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.100444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.100448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.100606 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.101290 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:21.101298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:21.101546 (+ 248us) client_negotiation.cc:769] Sending connection context
0923 20:59:21.101772 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 20:59:21.102246 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:59:21.110765 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354: Bootstrap starting.
I20260923 20:59:21.111169 23039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:21.116253 22890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:21.117141 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.094413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.0.97.62:38537)
0923 20:59:21.096984 (+ 2571us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.096990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.097009 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:59:21.097067 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.097075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.097184 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.097300 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.099229 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.100759 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.102744 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.103601 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.103745 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.109337 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.109344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.109352 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.109414 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.113402 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.114373 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.114379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.114382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.114543 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.115017 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.115023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.115028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.115444 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.115760 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.116150 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.116788 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2468,"thread_start_us":15046,"threads_started":1}
I20260923 20:59:21.117897 22890 log.cc:824] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:21.121667 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354: No bootstrap required, opened a new log
I20260923 20:59:21.129789 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.112136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44263 (local address 127.0.0.1:47018)
0923 20:59:21.112368 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:21.112392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:21.112588 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:21.113196 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:21.113202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:21.113743 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:21.114020 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.114035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.115338 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.115348 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.116680 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.116696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.117466 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.117471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.117668 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.118690 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:21.118715 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:21.122055 (+ 3340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:21.126237 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.126243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.126248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.126882 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.127463 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.127469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.127472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.127666 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.128712 (+ 1046us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:21.128717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:21.128907 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 20:59:21.129083 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 20:59:21.129415 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:59:21.130007 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.112150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.0.97.61:44263)
0923 20:59:21.112346 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.112351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.112471 (+ 120us) server_negotiation.cc:407] Connection header received
0923 20:59:21.112749 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.112755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.112847 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.112984 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.114202 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.115174 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.116846 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.117282 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.117434 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.122306 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.122313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.122318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.122366 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.126022 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.127031 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.127037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.127041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.127294 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.127811 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.127817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.127820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.128560 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.128738 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.129226 (+ 488us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.129508 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:21.132302 23039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:21.132596 22890 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.133028 22890 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7493aa985104d3498e3d0a9efcae354, State: Initialized, Role: FOLLOWER
I20260923 20:59:21.133656 22890 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.134702 23049 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.135202 23049 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:21.136059 22890 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:21.141135 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.119621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.0.97.61:44263)
0923 20:59:21.120196 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.120202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.120229 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:59:21.120290 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.120294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.120368 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.120487 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.123160 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.124499 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.126583 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.127268 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.127423 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.132905 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.132915 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.132923 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.133345 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.137172 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.138188 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.138194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.138198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.138351 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.138791 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.138797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.138800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.139177 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.139480 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.140140 (+ 660us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.140495 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":528,"threads_started":1}
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:59:21.148041 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:21.149580 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.133228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.0.97.60:45171)
0923 20:59:21.133411 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.133417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.133531 (+ 114us) server_negotiation.cc:407] Connection header received
0923 20:59:21.133693 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.133698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.133791 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.133927 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.134911 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.136074 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.137890 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.138845 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.139059 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.143481 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.143490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.143495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.143536 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.146648 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.147420 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.147425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.147429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.147569 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.147987 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.147995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.147998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.148364 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.148518 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.148912 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.149268 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":208,"server-negotiator.queue_time_us":53}
I20260923 20:59:21.149580 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.133113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36604)
0923 20:59:21.133438 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:21.133458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:21.133574 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:21.133974 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:21.133981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:21.134535 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:21.134753 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.134765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.136252 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.136256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.137667 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.137684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.139076 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.139083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.139416 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.140117 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:21.140140 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:21.143302 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:21.146805 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.146811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.146814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.147273 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.147702 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.147707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.147715 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.147887 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.148492 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:21.148497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:21.148631 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 20:59:21.148784 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 20:59:21.149142 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":26}
I20260923 20:59:21.152215 22865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c7493aa985104d3498e3d0a9efcae354"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37894
I20260923 20:59:21.152833 22865 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:59:21.153075 22865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:21.155989 23039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: Bootstrap starting.
I20260923 20:59:21.156493 22865 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.159217 22865 leader_election.cc:288] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [CANDIDATE]: Term 1 election: Requested vote from peers 0da6fca763c847419376e9f7c680f2ae (127.0.97.61:44263), 612a0fa714c547019ef09237cf3ed61a (127.0.97.60:45171)
I20260923 20:59:21.160493 23039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:21.162025 23039 log.cc:824] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: Log is configured to *not* fsync() on all Append() calls
W20260923 20:59:21.175453 22844 tablet.cc:2369] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:59:21.177290 23060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:59:21.177459 23060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:59:21.180955 23039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: No bootstrap required, opened a new log
I20260923 20:59:21.166682 22823 master_runner.cc:186] Error getting master registration for 127.0.97.61:44263: OK, instance_id { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" instance_seqno: 1790197160494048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:59:21.184382 22823 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:59:21.195523 23039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.196087 23039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612a0fa714c547019ef09237cf3ed61a, State: Initialized, Role: FOLLOWER
I20260923 20:59:21.196673 23039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.198203 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.199447 23039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:21.199749 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7493aa985104d3498e3d0a9efcae354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "612a0fa714c547019ef09237cf3ed61a"
I20260923 20:59:21.200285 23014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:21.206295 23014 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7493aa985104d3498e3d0a9efcae354 in term 1.
I20260923 20:59:21.207880 22842 leader_election.cc:302] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 612a0fa714c547019ef09237cf3ed61a, c7493aa985104d3498e3d0a9efcae354; no voters:
I20260923 20:59:21.208647 23049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:21.216120 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:21.221686 23049 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [term 1 LEADER]: Becoming Leader. State: Replica: c7493aa985104d3498e3d0a9efcae354, State: Running, Role: LEADER
I20260923 20:59:21.223434 23049 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.228977 22964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } attempt: 5
Sep 23 20:59:21 dist-test-slave-plcg krb5kdc[22810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:59:21.262696 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.230746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.97.60:45171)
0923 20:59:21.230969 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.230976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.230995 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:59:21.231060 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.231068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.231160 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.231264 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.232595 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.233702 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.235592 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.236303 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.236448 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.251694 (+ 15246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.251702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.251708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.251752 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.255963 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.260033 (+ 4070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.260042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.260048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.260262 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.260849 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.260854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.260857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.261263 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.261403 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.261955 (+ 552us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.262229 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:59:21.266474 23072 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7493aa985104d3498e3d0a9efcae354. Latest consensus state: current_term: 1 leader_uuid: "c7493aa985104d3498e3d0a9efcae354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.267093 23072 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:21.267982 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.229712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45171 (local address 127.0.0.1:36618)
0923 20:59:21.230279 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:21.230306 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:21.230461 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:21.231481 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:21.231489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:21.232122 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:21.232348 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.232364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.233917 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.233923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.235396 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:21.235414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.236727 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.236734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:21.236977 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.237864 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:21.237910 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:21.251439 (+ 13529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:21.259209 (+ 7770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.259215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.259220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.259804 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.260434 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:21.260442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:21.260448 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:21.260695 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:21.261467 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:21.261472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:21.261620 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 20:59:21.261769 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 20:59:21.267576 (+ 5807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260923 20:59:21.274536 22964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae: Bootstrap starting.
I20260923 20:59:21.277705 23076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:21.279287 22964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:21.280751 22964 log.cc:824] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:21.281001 23076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:59:21.286185 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.199386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.0.97.62:38537)
0923 20:59:21.199635 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.199642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.230884 (+ 31242us) server_negotiation.cc:407] Connection header received
0923 20:59:21.231272 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.231281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.231395 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.231523 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.246617 (+ 15094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.248158 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.251547 (+ 3389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.252302 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.252460 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.259270 (+ 6810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.259279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.259286 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.259336 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.266055 (+ 6719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.267083 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.267089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.267096 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.267268 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.283221 (+ 15953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.283228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.283231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.283724 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.284077 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.285436 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.285741 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260923 20:59:21.288420 22993 tablet.cc:2369] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:59:21.291118 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.199471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.97.61:44263)
0923 20:59:21.199723 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.199732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.250207 (+ 50475us) server_negotiation.cc:407] Connection header received
0923 20:59:21.250490 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.250498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.250592 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.250694 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.253137 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.254292 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.256078 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.256788 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.256953 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.259993 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.260002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.260008 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.260056 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.264093 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.282515 (+ 18422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.282521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.282525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.282704 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.288269 (+ 5565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.288277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.288281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.288729 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.289018 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.290464 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.290746 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:59:21.301731 23014 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [term 1 FOLLOWER]: Refusing update from remote peer c7493aa985104d3498e3d0a9efcae354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:59:21.303237 23072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:59:21.310297 22964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae: No bootstrap required, opened a new log
I20260923 20:59:21.314531 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.198757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36608 (local address 127.0.97.60:45171)
0923 20:59:21.198974 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.198981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.231877 (+ 32896us) server_negotiation.cc:407] Connection header received
0923 20:59:21.232120 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.232125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.232219 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.232294 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.292848 (+ 60554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.294042 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.296198 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.296984 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.297100 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.301500 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:21.301508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:21.301513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:21.301563 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:21.305521 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.306540 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.306546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.306550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.306754 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:21.312176 (+ 5422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:21.312182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:21.312186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:21.312653 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:21.312978 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.313761 (+ 783us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.314069 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:59:21.328493 22964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.329103 22964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da6fca763c847419376e9f7c680f2ae, State: Initialized, Role: FOLLOWER
I20260923 20:59:21.329751 22964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } }
I20260923 20:59:21.333283 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7493aa985104d3498e3d0a9efcae354. Latest consensus state: current_term: 1 leader_uuid: "c7493aa985104d3498e3d0a9efcae354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.333737 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:21.337152 23079 mvcc.cc:205] Tried to move back new op lower bound from 7332647572673851392 to 7332647572389576704. Current Snapshot: MvccSnapshot[applied={T|T < 7332647572673851392}]
I20260923 20:59:21.340363 23080 mvcc.cc:205] Tried to move back new op lower bound from 7332647572673851392 to 7332647572389576704. Current Snapshot: MvccSnapshot[applied={T|T < 7332647572673851392}]
I20260923 20:59:21.347297 23049 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7493aa985104d3498e3d0a9efcae354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.347761 23049 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:21.350090 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7493aa985104d3498e3d0a9efcae354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.350402 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:21.348518 23083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.351320 23083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:21.352840 22939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7493aa985104d3498e3d0a9efcae354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0da6fca763c847419376e9f7c680f2ae"
I20260923 20:59:21.353318 22939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:21.354614 23076 catalog_manager.cc:1347] Generated new cluster ID: 1a42029fe43b403c9219978f3faebaa7
I20260923 20:59:21.354727 23076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:21.356923 22939 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7493aa985104d3498e3d0a9efcae354 in term 1.
I20260923 20:59:21.357868 22938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [term 1 FOLLOWER]: Refusing update from remote peer c7493aa985104d3498e3d0a9efcae354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:59:21.360692 22964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:21.365718 23072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:59:21.391690 23076 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:21.394207 23076 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:21.410069 23083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7493aa985104d3498e3d0a9efcae354. Latest consensus state: current_term: 1 leader_uuid: "c7493aa985104d3498e3d0a9efcae354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.410449 23083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:21.410830 23049 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7493aa985104d3498e3d0a9efcae354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.410921 23096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7493aa985104d3498e3d0a9efcae354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7493aa985104d3498e3d0a9efcae354" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38537 } } peers { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44263 } } peers { permanent_uuid: "612a0fa714c547019ef09237cf3ed61a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45171 } } }
I20260923 20:59:21.411260 23049 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:21.411373 23096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:21.435385 23076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7493aa985104d3498e3d0a9efcae354: Generated new TSK 0
I20260923 20:59:21.436694 23076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:59:21.450989 23101 catalog_manager.cc:1259] Loaded cluster ID: 1a42029fe43b403c9219978f3faebaa7
I20260923 20:59:21.451117 23101 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: loading cluster ID for follower catalog manager: success
I20260923 20:59:21.455142 23101 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: acquiring CA information for follower catalog manager: success
I20260923 20:59:21.457861 23101 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 612a0fa714c547019ef09237cf3ed61a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:59:21.429194 22972 master_runner.cc:186] Error getting master registration for 127.0.97.61:44263: OK, instance_id { permanent_uuid: "0da6fca763c847419376e9f7c680f2ae" instance_seqno: 1790197160494048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:59:21.463474 22972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:59:21.467262 23107 catalog_manager.cc:1259] Loaded cluster ID: 1a42029fe43b403c9219978f3faebaa7
I20260923 20:59:21.467451 23107 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae: loading cluster ID for follower catalog manager: success
I20260923 20:59:21.471625 23107 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae: acquiring CA information for follower catalog manager: success
I20260923 20:59:21.474406 23107 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0da6fca763c847419376e9f7c680f2ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:59:21.603621 23048 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:21.583185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.0.97.61:44263)
0923 20:59:21.583371 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.583377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.600988 (+ 17611us) server_negotiation.cc:407] Connection header received
0923 20:59:21.601793 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.601801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.601947 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.602045 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.603325 (+ 1280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47032: BlockingRecv error: recv got EOF from 127.0.0.1:47032 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:59:21.605782 23043 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:21.576846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.0.97.62:38537)
0923 20:59:21.577145 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.577155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.600448 (+ 23293us) server_negotiation.cc:407] Connection header received
0923 20:59:21.604106 (+ 3658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.604113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.604209 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.604289 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.605496 (+ 1207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37910: BlockingRecv error: recv got EOF from 127.0.0.1:37910 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:59:21.609167 23045 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:21.596978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.97.60:45171)
0923 20:59:21.597209 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.597216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.606931 (+ 9715us) server_negotiation.cc:407] Connection header received
0923 20:59:21.607479 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.607486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.607578 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.607669 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:21.608851 (+ 1182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36624: BlockingRecv error: recv got EOF from 127.0.0.1:36624 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 20:59:21.659895 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.640102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.0.97.61:44263)
0923 20:59:21.640300 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.640306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.651065 (+ 10759us) server_negotiation.cc:407] Connection header received
0923 20:59:21.651653 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.651660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.651933 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.652115 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:21.652898 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.654121 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.656423 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.657177 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.657287 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.658169 (+ 882us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:21.658943 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:21.659079 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.659357 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.659472 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:59:21.714052 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.698013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.0.97.60:45171)
0923 20:59:21.698226 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.698233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.705226 (+ 6993us) server_negotiation.cc:407] Connection header received
0923 20:59:21.705806 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.705839 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.706073 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.706226 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:21.706960 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.708227 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.710555 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.711274 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.711389 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.712280 (+ 891us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:21.713055 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:21.713206 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.713532 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.713645 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:59:21.798909 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.757559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.0.97.61:44263)
0923 20:59:21.757781 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.757788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.776845 (+ 19057us) server_negotiation.cc:407] Connection header received
0923 20:59:21.777659 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.777668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.777771 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.777881 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:21.780062 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.781401 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.792648 (+ 11247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.793473 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.793596 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.796879 (+ 3283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:21.797050 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:21.797125 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.798459 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.798573 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:59:21.801715 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.750912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.0.97.62:38537)
0923 20:59:21.751168 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.751175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.776258 (+ 25083us) server_negotiation.cc:407] Connection header received
0923 20:59:21.778756 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.778764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.779015 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.779181 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:21.783920 (+ 4739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.785344 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.794314 (+ 8970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.795092 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.795200 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.797530 (+ 2330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:21.798081 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:21.798200 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.801211 (+ 3011us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.801328 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260923 20:59:21.808315 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22823
I20260923 20:59:21.809640 22885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:21.810038 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:21.770949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.0.97.60:45171)
0923 20:59:21.771197 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:21.771204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:21.788381 (+ 17177us) server_negotiation.cc:407] Connection header received
0923 20:59:21.788692 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:21.788699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:21.788797 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:21.788880 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:21.789669 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.790927 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.802639 (+ 11712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:21.803393 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:21.803522 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:21.804331 (+ 809us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:21.804492 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:21.804559 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:21.809526 (+ 4967us) server_negotiation.cc:299] Negotiation successful
0923 20:59:21.809653 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 20:59:21.992906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22823
I20260923 20:59:22.023324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22897
W20260923 20:59:22.023779 22914 connection.cc:535] server connection from 127.0.0.1:47006 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 20:59:22.024420 22991 connection.cc:535] server connection from 127.0.0.1:36594 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:59:22.024953 22959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:22.161088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22897
I20260923 20:59:22.181041 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 22972
I20260923 20:59:22.182533 23034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:22.292374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 22972
2026-09-23T20:59:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4224 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:59:22 dist-test-slave-plcg krb5kdc[23140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:59:22 dist-test-slave-plcg krb5kdc[23140](info): set up 2 sockets
Sep 23 20:59:22 dist-test-slave-plcg krb5kdc[23140](info): commencing operation
krb5kdc: starting...
W20260923 20:59:23.414404 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/test-user.keytab.
Sep 23 20:59:23 dist-test-slave-plcg krb5kdc[23140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:59:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 20:59:23.693044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41895
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41895,127.0.97.61:45791,127.0.97.60:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:59:24.041941 23153 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:59:24.042335 23153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:24.042455 23153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:24.042589 23153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:24.050419 23153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:24.050523 23153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:24.050613 23153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:24.050707 23153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:24.050817 23153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:24.061455 23153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41895,127.0.97.61:45791,127.0.97.60:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:24.062881 23153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:24.064539 23153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:24.077337 23159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:24.077287 23158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:24.077382 23161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:24.078276 23153 server_base.cc:1034] running on GCE node
I20260923 20:59:24.079126 23153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:24.080875 23153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:24.082176 23153 hybrid_clock.cc:648] HybridClock initialized: now 1790197164082108 us; error 70 us; skew 500 ppm
I20260923 20:59:24.082567 23153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:24 dist-test-slave-plcg krb5kdc[23140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:24.085956 23153 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 20:59:24.087560 23153 webserver.cc:460] Webserver started at http://127.0.97.62:34301/ using document root <none> and password file <none>
I20260923 20:59:24.088325 23153 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:24.088467 23153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:24.088822 23153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:24.092108 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data/instance:
uuid: "9f51899045844302ae1bd14777ced794"
format_stamp: "Formatted at 2026-09-23 20:59:24 on dist-test-slave-plcg"
server_key: "e072e5aaf72633da642808c7253dc089"
server_key_iv: "5f5b4c90bd9f9eb113bc3a5231558a1e"
server_key_version: "encryptionkey@0"
I20260923 20:59:24.092984 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/wal/instance:
uuid: "9f51899045844302ae1bd14777ced794"
format_stamp: "Formatted at 2026-09-23 20:59:24 on dist-test-slave-plcg"
server_key: "e072e5aaf72633da642808c7253dc089"
server_key_iv: "5f5b4c90bd9f9eb113bc3a5231558a1e"
server_key_version: "encryptionkey@0"
I20260923 20:59:24.098852 23153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:59:24.102720 23168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:24.104301 23153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:59:24.104542 23153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/wal
uuid: "9f51899045844302ae1bd14777ced794"
format_stamp: "Formatted at 2026-09-23 20:59:24 on dist-test-slave-plcg"
server_key: "e072e5aaf72633da642808c7253dc089"
server_key_iv: "5f5b4c90bd9f9eb113bc3a5231558a1e"
server_key_version: "encryptionkey@0"
I20260923 20:59:24.104802 23153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:24.140693 23153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:24.145493 23153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:24.145840 23153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:24.165635 23153 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41895
I20260923 20:59:24.165655 23219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41895 every 8 connection(s)
I20260923 20:59:24.167209 23153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/data/info.pb
I20260923 20:59:24.171074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 23153
I20260923 20:59:24.171279 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-0/wal/instance
I20260923 20:59:24.171278 23220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:24.171752 388 external_mini_cluster.cc:1428] Setting key ca58cf80dd0c19f04e0222ed0f17eaa3
I20260923 20:59:24.178484 23220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:24 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260923 20:59:24.206197 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.182592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.97.62:41895)
0923 20:59:24.183634 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:24.183655 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:24.184444 (+ 789us) server_negotiation.cc:407] Connection header received
0923 20:59:24.186764 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:24.186775 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:24.187148 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:24.187400 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:24.188662 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.190724 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.192406 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.193542 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.193845 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.197907 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:24.197918 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:24.197932 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:24.197968 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:24.201392 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.202284 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.202295 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.202303 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.202465 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.202886 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.202892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.202895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.203441 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:24.203662 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:24.204224 (+ 562us) server_negotiation.cc:299] Negotiation successful
0923 20:59:24.204637 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":725,"thread_start_us":452,"threads_started":1}
I20260923 20:59:24.206221 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.182592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41895 (local address 127.0.0.1:59156)
0923 20:59:24.184220 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.184272 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:24.186278 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:24.187436 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:24.187442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:24.188039 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:24.188391 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.188429 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.190981 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.190987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.192195 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.192210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.193969 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.193982 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.194130 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.194889 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:24.194913 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:24.197566 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:24.201564 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.201579 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.201594 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.202127 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.202584 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.202591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.202596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.202766 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.203577 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:24.203590 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:24.203829 (+ 239us) client_negotiation.cc:769] Sending connection context
0923 20:59:24.204089 (+ 260us) client_negotiation.cc:240] Negotiation successful
0923 20:59:24.204771 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"mutex_wait_us":165,"thread_start_us":1187,"threads_started":1}
I20260923 20:59:24.209527 23220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:24.210830 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.210380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39458)
0923 20:59:24.210570 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.210704 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:59:24.211130 23171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
I20260923 20:59:24.212919 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.212534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39470)
0923 20:59:24.212692 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.212780 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:59:24.213153 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45791: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 20:59:24.234583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45791
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41895,127.0.97.61:45791,127.0.97.60:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:59:24.262485 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } attempt: 1
I20260923 20:59:24.263684 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.263221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39480)
0923 20:59:24.263445 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.263533 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:59:24.264719 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.264340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39494)
0923 20:59:24.264492 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.264582 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:59:24.264926 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45791: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
I20260923 20:59:24.333199 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } attempt: 2
I20260923 20:59:24.334540 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.334023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39500)
0923 20:59:24.334303 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.334394 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:59:24.335621 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.335237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39510)
0923 20:59:24.335388 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.335479 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:59:24.335908 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45791: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
I20260923 20:59:24.427248 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } attempt: 3
I20260923 20:59:24.428435 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.427975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39516)
0923 20:59:24.428185 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.428279 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:59:24.429502 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.429131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39530)
0923 20:59:24.429281 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.429355 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:59:24.429769 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45791: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
I20260923 20:59:24.573115 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } attempt: 4
I20260923 20:59:24.574482 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.573907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39544)
0923 20:59:24.574257 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.574345 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260923 20:59:24.575691 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.575238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39552)
0923 20:59:24.575389 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.575495 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:59:24.575917 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45791: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45791: connect: Connection refused (error 111)
W20260923 20:59:24.583722 23227 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:59:24.584153 23227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:24.584286 23227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:24.584430 23227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:24.592248 23227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:24.592394 23227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:24.592506 23227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:24.592600 23227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:24.592712 23227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:24.603397 23227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41895,127.0.97.61:45791,127.0.97.60:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:24.604820 23227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:24.606226 23227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:24.619405 23232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:24.619376 23233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:24.619685 23235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:24.620889 23227 server_base.cc:1034] running on GCE node
I20260923 20:59:24.621743 23227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:24.623339 23227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:24.624641 23227 hybrid_clock.cc:648] HybridClock initialized: now 1790197164624549 us; error 67 us; skew 500 ppm
I20260923 20:59:24.625041 23227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:24 dist-test-slave-plcg krb5kdc[23140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:24.628533 23227 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 20:59:24.630406 23227 webserver.cc:460] Webserver started at http://127.0.97.61:38237/ using document root <none> and password file <none>
I20260923 20:59:24.631214 23227 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:24.631356 23227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:24.631701 23227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:24.635097 23227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data/instance:
uuid: "5787a807cf5347c6963ef990e248fe99"
format_stamp: "Formatted at 2026-09-23 20:59:24 on dist-test-slave-plcg"
server_key: "5ab1b7be39959251f27004c3ef684841"
server_key_iv: "699f63af676dc06cc51b2875f846ed45"
server_key_version: "encryptionkey@0"
I20260923 20:59:24.635972 23227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/wal/instance:
uuid: "5787a807cf5347c6963ef990e248fe99"
format_stamp: "Formatted at 2026-09-23 20:59:24 on dist-test-slave-plcg"
server_key: "5ab1b7be39959251f27004c3ef684841"
server_key_iv: "699f63af676dc06cc51b2875f846ed45"
server_key_version: "encryptionkey@0"
I20260923 20:59:24.641904 23227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:59:24.645798 23242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:24.647385 23227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:59:24.647621 23227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/wal
uuid: "5787a807cf5347c6963ef990e248fe99"
format_stamp: "Formatted at 2026-09-23 20:59:24 on dist-test-slave-plcg"
server_key: "5ab1b7be39959251f27004c3ef684841"
server_key_iv: "699f63af676dc06cc51b2875f846ed45"
server_key_version: "encryptionkey@0"
I20260923 20:59:24.647883 23227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:24.678066 23227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:24.682771 23227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:24.683087 23227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:24.703199 23227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45791
I20260923 20:59:24.703218 23293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45791 every 8 connection(s)
I20260923 20:59:24.704661 23227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/data/info.pb
I20260923 20:59:24.709023 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:24.712323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 23227
I20260923 20:59:24.712586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-1/wal/instance
I20260923 20:59:24.713052 388 external_mini_cluster.cc:1428] Setting key 709b9d9413bfb87bd85a2ee9c542626b
I20260923 20:59:24.716856 23294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:24 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260923 20:59:24.742967 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.719805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.0.97.62:41895)
0923 20:59:24.720357 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:24.720361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:24.721705 (+ 1344us) server_negotiation.cc:407] Connection header received
0923 20:59:24.723947 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:24.723951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:24.724028 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:24.724133 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:24.726512 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.727260 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.729914 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.730358 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.730474 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.735281 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:24.735286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:24.735290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:24.735339 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:24.738982 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.739935 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.739944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.739950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.740137 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.740802 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.740810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.740815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.741297 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:24.741495 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:24.742392 (+ 897us) server_negotiation.cc:299] Negotiation successful
0923 20:59:24.742595 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":318,"threads_started":1}
I20260923 20:59:24.744029 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.720471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41895 (local address 127.0.0.1:59160)
0923 20:59:24.721485 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.721541 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:24.723658 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:24.724422 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:24.724437 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:24.725313 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:24.726205 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.726235 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.727464 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.727473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.729637 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.729658 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.730641 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.730651 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.731035 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.731861 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:24.731915 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:24.734981 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:24.739198 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.739207 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.739223 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.739750 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.740283 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.740287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.740289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.740614 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.741460 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:24.741468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:24.741989 (+ 521us) client_negotiation.cc:769] Sending connection context
0923 20:59:24.742244 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 20:59:24.742692 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":403,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 20:59:24.746024 23294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:24 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:24.764094 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.746999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39564)
0923 20:59:24.747235 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.747260 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:24.747532 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:24.748474 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:24.748477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:24.748875 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:24.749011 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.749019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.751137 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.751141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.752173 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.752189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.753551 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.753556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.753721 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.754433 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:24.754451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:24.756695 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:24.761470 (+ 4775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.761475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.761478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.761970 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.762407 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.762411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.762413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.762535 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.763322 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:24.763325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:24.763462 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 20:59:24.763585 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 20:59:24.763810 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":5}
I20260923 20:59:24.764591 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.747077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.0.97.61:45791)
0923 20:59:24.747915 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:24.747925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:24.747960 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:59:24.748024 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:24.748034 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:24.748299 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:24.748471 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:24.749187 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.750944 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.752353 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.753381 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.753548 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.757008 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:24.757028 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:24.757052 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:24.757101 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:24.761283 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.762120 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.762132 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.762139 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.762293 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.762642 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.762649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.762655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.763188 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:24.763426 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:24.763761 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 20:59:24.764076 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":635,"thread_start_us":492,"threads_started":1}
I20260923 20:59:24.767424 23294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:24.768852 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.768298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33348)
0923 20:59:24.768514 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.768686 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260923 20:59:24.769166 23246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
I20260923 20:59:24.770929 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.770536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33352)
0923 20:59:24.770729 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.770809 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:59:24.771157 23294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 20:59:24.786401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
/tmp/dist-test-taskf1GrEz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44033
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41895,127.0.97.61:45791,127.0.97.60:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:59:24.820505 23294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 1
I20260923 20:59:24.821697 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.821223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33358)
0923 20:59:24.821449 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.821537 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:59:24.822800 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.822445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33364)
0923 20:59:24.822613 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.822688 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:59:24.823089 23294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
I20260923 20:59:24.875200 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } attempt: 5
Sep 23 20:59:24 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:24.889607 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.876193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.97.61:45791)
0923 20:59:24.876416 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:24.876422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:24.876444 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:59:24.876664 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:24.876668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:24.876737 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:24.876850 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:24.877761 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.878540 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.879768 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.880242 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.880384 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.884049 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:24.884054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:24.884057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:24.884093 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:24.886823 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.887586 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.887593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.887600 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.887736 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:24.888142 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:24.888148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:24.888151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:24.888451 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:24.888635 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:24.889164 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 20:59:24.889352 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 20:59:24.889940 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.876062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39574)
0923 20:59:24.876302 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.876326 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:24.876470 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:24.876892 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:24.876898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:24.877399 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:24.877594 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.877605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.878701 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.878708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.879599 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:24.879613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:24.880418 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:24.880425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:24.880662 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:24.881375 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:24.881394 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:24.883789 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:24.887007 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.887014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.887018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.887435 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.887860 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:24.887865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:24.887869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:24.888023 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:24.888657 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:24.888663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:24.888862 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 20:59:24.889049 (+ 187us) client_negotiation.cc:240] Negotiation successful
0923 20:59:24.889409 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:59:24.891093 23220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:24.891333 23294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 2
I20260923 20:59:24.892302 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.891833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33372)
0923 20:59:24.892087 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.892164 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260923 20:59:24.892299 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.891829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33376)
0923 20:59:24.892029 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.892131 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:59:24.893404 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.892977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33384)
0923 20:59:24.893140 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.893255 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 20:59:24.893505 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.893110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33392)
0923 20:59:24.893282 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.893360 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 20:59:24.893775 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
W20260923 20:59:24.893915 23294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
I20260923 20:59:24.945065 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 1
I20260923 20:59:24.946345 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.945864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33400)
0923 20:59:24.946074 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.946174 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:59:24.947434 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.947095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33406)
0923 20:59:24.947257 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.947323 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:59:24.947659 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
I20260923 20:59:24.985208 23294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 3
I20260923 20:59:24.986789 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.986088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33408)
0923 20:59:24.986382 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.986534 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:59:24.988039 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:24.987508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33410)
0923 20:59:24.987761 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:24.987859 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260923 20:59:24.988330 23294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
I20260923 20:59:25.015898 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 2
I20260923 20:59:25.017115 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.016614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33420)
0923 20:59:25.016857 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.016957 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 20:59:25.018277 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.017932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33422)
0923 20:59:25.018095 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.018165 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:59:25.018492 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
I20260923 20:59:25.124732 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 3
I20260923 20:59:25.126060 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.125509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33438)
0923 20:59:25.125723 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.125868 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 20:59:25.127213 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.126783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33452)
0923 20:59:25.127030 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.127102 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260923 20:59:25.127418 23220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
I20260923 20:59:25.131559 23294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 4
I20260923 20:59:25.132763 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.132207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33460)
0923 20:59:25.132461 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.132568 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:59:25.134004 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.133555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33462)
0923 20:59:25.133733 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.133851 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 20:59:25.134270 23294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44033: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44033: connect: Connection refused (error 111)
W20260923 20:59:25.145057 23302 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:59:25.145406 23302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:59:25.145485 23302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:59:25.145576 23302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:59:25.154162 23302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:59:25.154269 23302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:59:25.154327 23302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:59:25.154378 23302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:59:25.154441 23302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:59:25.164918 23302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41895,127.0.97.61:45791,127.0.97.60:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 20:59:25.166491 23302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:59:25.167936 23302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:59:25.180934 23308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:25.180871 23307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:59:25.181442 23310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:59:25.181967 23302 server_base.cc:1034] running on GCE node
I20260923 20:59:25.182904 23302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:59:25.184484 23302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:59:25.185791 23302 hybrid_clock.cc:648] HybridClock initialized: now 1790197165185700 us; error 68 us; skew 500 ppm
I20260923 20:59:25.186232 23302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:59:25.189668 23302 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 20:59:25.191365 23302 webserver.cc:460] Webserver started at http://127.0.97.60:38703/ using document root <none> and password file <none>
I20260923 20:59:25.192173 23302 fs_manager.cc:359] Metadata directory not provided
I20260923 20:59:25.192319 23302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:59:25.192672 23302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:59:25.195905 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data/instance:
uuid: "3627a5be2307411e87ca6d1fe20b97c6"
format_stamp: "Formatted at 2026-09-23 20:59:25 on dist-test-slave-plcg"
server_key: "d98bb6606132df4858977b0d11cbc5b8"
server_key_iv: "098c92db7b9f2ad521f3f71f637883cd"
server_key_version: "encryptionkey@0"
I20260923 20:59:25.196796 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/wal/instance:
uuid: "3627a5be2307411e87ca6d1fe20b97c6"
format_stamp: "Formatted at 2026-09-23 20:59:25 on dist-test-slave-plcg"
server_key: "d98bb6606132df4858977b0d11cbc5b8"
server_key_iv: "098c92db7b9f2ad521f3f71f637883cd"
server_key_version: "encryptionkey@0"
I20260923 20:59:25.202704 23302 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:59:25.206471 23317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:59:25.207908 23302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:59:25.208124 23302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/wal
uuid: "3627a5be2307411e87ca6d1fe20b97c6"
format_stamp: "Formatted at 2026-09-23 20:59:25 on dist-test-slave-plcg"
server_key: "d98bb6606132df4858977b0d11cbc5b8"
server_key_iv: "098c92db7b9f2ad521f3f71f637883cd"
server_key_version: "encryptionkey@0"
I20260923 20:59:25.208410 23302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:59:25.237540 23302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:59:25.241957 23302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:59:25.242280 23302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:59:25.261945 23302 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44033
I20260923 20:59:25.261965 23368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44033 every 8 connection(s)
I20260923 20:59:25.263437 23302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/data/info.pb
I20260923 20:59:25.263978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu as pid 23302
I20260923 20:59:25.264210 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf1GrEz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790196790989944-388-0/minicluster-data/master-2/wal/instance
I20260923 20:59:25.264892 388 external_mini_cluster.cc:1428] Setting key f3a19c4a4b18f56272bd51273be1ef92
I20260923 20:59:25.268055 23369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:59:25.277022 23369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:59:25.290915 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.266742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.0.97.60:44033)
0923 20:59:25.267563 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.267575 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.267618 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:59:25.269388 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.269410 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.269924 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.270300 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.272175 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.273867 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.275787 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.276514 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.276686 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.282207 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.282243 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.282263 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.282345 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.286098 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.286979 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.286985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.286992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.287166 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.287618 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.287623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.287626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.288391 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.288734 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.289183 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.289654 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":369,"threads_started":1}
I20260923 20:59:25.298062 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.280563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.0.97.62:41895)
0923 20:59:25.282095 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.282101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.282124 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:59:25.282362 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.282368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.282460 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.282597 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.283874 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.285092 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.286709 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.287279 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.287418 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.291393 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.291398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.291401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.291440 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.294776 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.295755 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.295763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.295768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.295956 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.296360 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.296366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.296369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.296746 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.296906 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.297450 (+ 544us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.297637 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":278,"threads_started":1}
I20260923 20:59:25.298224 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.280607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41895 (local address 127.0.0.1:59170)
0923 20:59:25.281870 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.281955 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:25.282218 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:25.282630 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:25.282636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:25.283368 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:25.283687 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.283697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.285285 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.285289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.286532 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.286546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.287467 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.287471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.287655 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.288306 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:25.288334 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:25.291132 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:25.294924 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.294939 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.294946 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.295569 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.296083 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.296086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.296088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.296224 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.296890 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:25.296899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:25.297123 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 20:59:25.297335 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 20:59:25.297696 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"spinlock_wait_cycles":21760,"thread_start_us":525,"threads_started":1}
I20260923 20:59:25.299965 23369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 20:59:25.304625 23220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 4
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:25.312935 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.294152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.0.97.62:41895)
0923 20:59:25.294783 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.294787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.294807 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:59:25.294878 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.294883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.294959 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.295083 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.296611 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.297918 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.299466 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.300025 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.300188 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.305371 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.305378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.305383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.305430 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.309038 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.310037 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.310044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.310048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.310234 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.310738 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.310747 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.310753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.311163 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.311447 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.312130 (+ 683us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.312382 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"spinlock_wait_cycles":30080,"thread_start_us":295,"threads_started":1}
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:59:25.316705 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.300850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45791 (local address 127.0.0.1:39586)
0923 20:59:25.301023 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.301042 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:25.301144 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:25.301544 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:25.301548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:25.301984 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:25.302130 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.302139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.303617 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.303620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.304675 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.304692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.305741 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.305746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.305979 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.306715 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:25.306740 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:25.309468 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:25.313576 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.313581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.313585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.314171 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.314691 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.314696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.314699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.314887 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.315644 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:25.315652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:25.315839 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 20:59:25.315991 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 20:59:25.316298 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:59:25.317728 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.300925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.0.97.61:45791)
0923 20:59:25.301114 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.301119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.301135 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:59:25.301290 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.301295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.301399 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.301510 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.302290 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.303448 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.304851 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.305512 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.305638 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.309695 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.309705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.309711 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.309759 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.313374 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.314349 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.314356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.314359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.314536 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.315031 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.315035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.315039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.315429 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.315587 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.317060 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.317310 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:59:25.322096 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.305414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.0.97.60:44033)
0923 20:59:25.305622 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.305628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.305647 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:59:25.305872 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.305878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.305970 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.306072 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.307619 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.308805 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.310835 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.311476 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.311890 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.315556 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.315561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.315566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.315605 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.319035 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.319745 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.319751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.319755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.319921 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.320340 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.320346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.320350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.320705 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.320856 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.321416 (+ 560us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.321705 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:59:25.322721 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.305264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33470)
0923 20:59:25.305560 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.305581 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:25.305703 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:25.306144 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:25.306150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:25.307205 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:25.307433 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.307446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.308972 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.308979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.310604 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.310622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.311614 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.311620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.311877 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.312476 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:25.312494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:25.315336 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:25.319194 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.319200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.319204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.319614 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.320045 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.320050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.320054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.320224 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.320868 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:25.320877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:25.321107 (+ 230us) client_negotiation.cc:769] Sending connection context
0923 20:59:25.321315 (+ 208us) client_negotiation.cc:240] Negotiation successful
0923 20:59:25.321761 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 20:59:25.329692 23369 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:59:25.331278 23220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794: Bootstrap starting.
I20260923 20:59:25.335269 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.315450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.0.97.61:45791)
0923 20:59:25.318007 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.318015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.318032 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:59:25.318090 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.318096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.318178 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.318286 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.320455 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.321417 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.322999 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.323686 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.323834 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.328372 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.328381 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.328387 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.328447 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.332116 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.333060 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.333066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.333070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.333253 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.333679 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.333684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.333688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.334086 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.334375 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.334663 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.334906 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2442,"thread_start_us":12975,"threads_started":1}
I20260923 20:59:25.335944 23220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:25.337460 23220 log.cc:824] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:59:25.340632 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:59:25.341761 23220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794: No bootstrap required, opened a new log
I20260923 20:59:25.344030 23195 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9f51899045844302ae1bd14777ced794"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59182
I20260923 20:59:25.347343 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.330600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33474)
0923 20:59:25.330835 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.330856 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:25.330977 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:25.331519 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:25.331524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:25.332021 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:25.332236 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.332251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.333306 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.333313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.334712 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.334731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.335934 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.335941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.336182 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.336872 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:25.336900 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:25.340456 (+ 3556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:25.344408 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.344414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.344419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.344979 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.345394 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.345399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.345403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.345578 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.346253 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:25.346258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:25.346420 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 20:59:25.346588 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 20:59:25.346937 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 20:59:25.348222 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.330935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.97.60:44033)
0923 20:59:25.331121 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.331127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.331146 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:59:25.331197 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.331201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.331281 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.331391 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.332355 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.333151 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.334873 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.335532 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.335630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.340665 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.340672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.340677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.340719 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.344245 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.345147 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.345151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.345153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.345279 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.345688 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.345691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.345693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.346127 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.346260 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.347648 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.347871 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:59:25.352414 23220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
I20260923 20:59:25.352901 23220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f51899045844302ae1bd14777ced794, State: Initialized, Role: FOLLOWER
I20260923 20:59:25.353464 23220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
I20260923 20:59:25.354434 23369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: Bootstrap starting.
I20260923 20:59:25.354600 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.355052 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:25.356004 23220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:25.356072 23195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:59:25.356314 23195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:25.359262 23369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:25.359350 23195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
I20260923 20:59:25.361140 23369 log.cc:824] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:25.362269 23195 leader_election.cc:288] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [CANDIDATE]: Term 1 election: Requested vote from peers 5787a807cf5347c6963ef990e248fe99 (127.0.97.61:45791), 3627a5be2307411e87ca6d1fe20b97c6 (127.0.97.60:44033)
I20260923 20:59:25.365525 23369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: No bootstrap required, opened a new log
I20260923 20:59:25.380936 23369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
I20260923 20:59:25.381580 23369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3627a5be2307411e87ca6d1fe20b97c6, State: Initialized, Role: FOLLOWER
I20260923 20:59:25.382347 23369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
I20260923 20:59:25.384286 23153 master_runner.cc:186] Error getting master registration for 127.0.97.61:45791: OK, instance_id { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" instance_seqno: 1790197164697395 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:59:25.396831 23153 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 20:59:25.397298 23394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:59:25.397490 23394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:59:25.398283 23399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.398733 23399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 20:59:25.398852 23174 tablet.cc:2369] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:59:25.399751 23369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:25.400159 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f51899045844302ae1bd14777ced794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3627a5be2307411e87ca6d1fe20b97c6"
I20260923 20:59:25.401014 23344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:25.405129 23344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f51899045844302ae1bd14777ced794 in term 1.
I20260923 20:59:25.406116 23172 leader_election.cc:302] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3627a5be2307411e87ca6d1fe20b97c6, 9f51899045844302ae1bd14777ced794; no voters:
I20260923 20:59:25.406656 23378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:59:25.407408 23378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [term 1 LEADER]: Becoming Leader. State: Replica: 9f51899045844302ae1bd14777ced794, State: Running, Role: LEADER
I20260923 20:59:25.408066 23378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
I20260923 20:59:25.422569 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.368009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.0.97.61:45791)
0923 20:59:25.368257 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.368266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.379405 (+ 11139us) server_negotiation.cc:407] Connection header received
0923 20:59:25.379959 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.379966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.380067 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.380151 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.385680 (+ 5529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.386866 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.396722 (+ 9856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.397333 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.397477 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.410748 (+ 13271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.410757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.410762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.410821 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.415884 (+ 5063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.417637 (+ 1753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.417644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.417647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.417978 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.419044 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.419052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.419056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.419709 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.420110 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.421423 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.421912 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:59:25.426667 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f51899045844302ae1bd14777ced794. Latest consensus state: current_term: 1 leader_uuid: "9f51899045844302ae1bd14777ced794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.427874 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:25.429504 23405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:59:25.433354 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.367940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.0.97.62:41895)
0923 20:59:25.368162 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.368168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.391095 (+ 22927us) server_negotiation.cc:407] Connection header received
0923 20:59:25.392866 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.392873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.392980 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.393079 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.405620 (+ 12541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.406802 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.410612 (+ 3810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.411286 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.411445 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.422774 (+ 11329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.422784 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.422791 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.422842 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.427487 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.430656 (+ 3169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.430662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.430665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.430842 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.431367 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.431374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.431376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.431799 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.432127 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.432730 (+ 603us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.432995 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:59:25.433477 23294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } attempt: 5
I20260923 20:59:25.434478 23405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 20:59:25 dist-test-slave-plcg krb5kdc[23140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 20:59:25.452979 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.369009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.0.97.60:44033)
0923 20:59:25.369293 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.369303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.404648 (+ 35345us) server_negotiation.cc:407] Connection header received
0923 20:59:25.405173 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.405181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.405275 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.405352 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.412846 (+ 7494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.414010 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.418668 (+ 4658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.419433 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.419539 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.442469 (+ 22930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.442475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.442480 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.442526 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.448549 (+ 6023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.449738 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.449746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.449753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.449942 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.450473 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.450478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.450481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.450907 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.451196 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.451927 (+ 731us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.452279 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 20:59:25.460692 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.434120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44033 (local address 127.0.0.1:33492)
0923 20:59:25.434542 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0923 20:59:25.434562 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:59:25.434675 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:59:25.436676 (+ 2001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:59:25.436682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:59:25.437189 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:59:25.437423 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.437437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.440583 (+ 3146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.440600 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.441979 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:59:25.441999 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.443821 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.443828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:59:25.444049 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.444648 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:59:25.444674 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:59:25.447687 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:59:25.457018 (+ 9331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.457024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.457029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.457740 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.458566 (+ 826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:59:25.458575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:59:25.458580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:59:25.458783 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:59:25.459617 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:59:25.459622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:59:25.459798 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 20:59:25.459944 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 20:59:25.460275 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260923 20:59:25.462167 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:25.434893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.0.97.60:44033)
0923 20:59:25.436054 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.436061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.436088 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:59:25.436204 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.436210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.436382 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.437647 (+ 1265us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.438218 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.440342 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.442277 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:25.443555 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:25.444458 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:25.447999 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:59:25.448007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:59:25.448039 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:59:25.448125 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:59:25.456725 (+ 8600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.458054 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.458062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.458095 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.458364 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:59:25.458964 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:59:25.458971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:59:25.458975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:59:25.459439 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:59:25.461281 (+ 1842us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:25.461417 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 20:59:25.461687 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":476,"threads_started":1}
I20260923 20:59:25.469480 23294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: Bootstrap starting.
I20260923 20:59:25.472719 23415 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:59:25.474699 23294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: Neither blocks nor log segments found. Creating new log.
I20260923 20:59:25.476219 23294 log.cc:824] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: Log is configured to *not* fsync() on all Append() calls
I20260923 20:59:25.481078 23294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: No bootstrap required, opened a new log
I20260923 20:59:25.481386 23344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [term 1 FOLLOWER]: Refusing update from remote peer 9f51899045844302ae1bd14777ced794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:59:25.482492 23401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:59:25.492936 23294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
I20260923 20:59:25.493413 23294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5787a807cf5347c6963ef990e248fe99, State: Initialized, Role: FOLLOWER
I20260923 20:59:25.494055 23294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } }
W20260923 20:59:25.494590 23323 tablet.cc:2369] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:59:25.486191 23302 master_runner.cc:186] Error getting master registration for 127.0.97.61:45791: OK, instance_id { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" instance_seqno: 1790197164697395 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:59:25.500692 23302 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:59:25.501133 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.501597 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:25.503641 23294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:59:25.503746 23269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f51899045844302ae1bd14777ced794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5787a807cf5347c6963ef990e248fe99"
I20260923 20:59:25.504395 23269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:59:25.510814 23269 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f51899045844302ae1bd14777ced794 in term 1.
I20260923 20:59:25.511751 23268 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [term 1 FOLLOWER]: Refusing update from remote peer 9f51899045844302ae1bd14777ced794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:59:25.513710 23401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:59:25.513891 23399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f51899045844302ae1bd14777ced794. Latest consensus state: current_term: 1 leader_uuid: "9f51899045844302ae1bd14777ced794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.514173 23399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:25.518015 23399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f51899045844302ae1bd14777ced794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.518364 23399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:25.537155 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f51899045844302ae1bd14777ced794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.537492 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:25.538450 23405 catalog_manager.cc:1347] Generated new cluster ID: 5bfaa38ed6074ec7aed585364a3ef131
I20260923 20:59:25.538583 23405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:59:25.538805 23418 mvcc.cc:205] Tried to move back new op lower bound from 7332647589799305216 to 7332647589517078528. Current Snapshot: MvccSnapshot[applied={T|T < 7332647589799305216}]
I20260923 20:59:25.544745 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f51899045844302ae1bd14777ced794. Latest consensus state: current_term: 1 leader_uuid: "9f51899045844302ae1bd14777ced794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.545581 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:59:25.549871 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f51899045844302ae1bd14777ced794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.550215 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794 [sys.catalog]: This master's current role is: LEADER
I20260923 20:59:25.553565 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f51899045844302ae1bd14777ced794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51899045844302ae1bd14777ced794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41895 } } peers { permanent_uuid: "5787a807cf5347c6963ef990e248fe99" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45791 } } peers { permanent_uuid: "3627a5be2307411e87ca6d1fe20b97c6" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44033 } } }
I20260923 20:59:25.554427 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 20:59:25.560942 23434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:59:25.561164 23434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:59:25.581912 23405 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:59:25.584038 23405 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:59:25.613996 23405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f51899045844302ae1bd14777ced794: Generated new TSK 0
I20260923 20:59:25.615340 23405 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:59:25.712512 23375 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.703358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.0.97.62:41895)
0923 20:59:25.703552 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.703558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.710667 (+ 7109us) server_negotiation.cc:407] Connection header received
0923 20:59:25.710751 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.710757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.710849 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.711011 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.712190 (+ 1179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59196: BlockingRecv error: recv got EOF from 127.0.0.1:59196 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 20:59:25.715510 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.710336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.0.97.61:45791)
0923 20:59:25.710580 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.710587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.713445 (+ 2858us) server_negotiation.cc:407] Connection header received
0923 20:59:25.713945 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.713952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.714042 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.714145 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.715213 (+ 1068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39622: BlockingRecv error: recv got EOF from 127.0.0.1:39622 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 20:59:25.718398 23407 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.716466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.0.97.60:44033)
0923 20:59:25.716676 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.716684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.716824 (+ 140us) server_negotiation.cc:407] Connection header received
0923 20:59:25.717004 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.717010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.717101 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.717212 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.718150 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33496: BlockingRecv error: recv got EOF from 127.0.0.1:33496 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:59:25.747605 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.744627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.0.97.61:45791)
0923 20:59:25.744821 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.744828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.745344 (+ 516us) server_negotiation.cc:407] Connection header received
0923 20:59:25.746117 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.746125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.746272 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.746380 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.747285 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39628: BlockingRecv error: recv got EOF from 127.0.0.1:39628 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:59:25.756053 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.753569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.0.97.61:45791)
0923 20:59:25.753757 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.753764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.754431 (+ 667us) server_negotiation.cc:407] Connection header received
0923 20:59:25.754637 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.754643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.754743 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.754857 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.755795 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39642: BlockingRecv error: recv got EOF from 127.0.0.1:39642 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 20:59:25.772197 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.762269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.0.97.61:45791)
0923 20:59:25.762457 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.762463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.770538 (+ 8075us) server_negotiation.cc:407] Connection header received
0923 20:59:25.770778 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.770784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.770900 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.771029 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.771927 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39646: BlockingRecv error: recv got EOF from 127.0.0.1:39646 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 20:59:25.786389 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.783950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.0.97.61:45791)
0923 20:59:25.784202 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.784209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.784796 (+ 587us) server_negotiation.cc:407] Connection header received
0923 20:59:25.785003 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.785009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.785125 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.785236 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.786109 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39662: BlockingRecv error: recv got EOF from 127.0.0.1:39662 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260923 20:59:25.800587 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.798236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.97.61:45791)
0923 20:59:25.798437 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.798444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.798963 (+ 519us) server_negotiation.cc:407] Connection header received
0923 20:59:25.799169 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.799175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.799299 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.799428 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.800336 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39670: BlockingRecv error: recv got EOF from 127.0.0.1:39670 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 20:59:25.828539 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.826066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.0.97.61:45791)
0923 20:59:25.826266 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.826273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.826837 (+ 564us) server_negotiation.cc:407] Connection header received
0923 20:59:25.827060 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.827068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.827189 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.827312 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.828240 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39686: BlockingRecv error: recv got EOF from 127.0.0.1:39686 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 20:59:25.867455 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.864895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.0.97.61:45791)
0923 20:59:25.865125 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.865133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.865686 (+ 553us) server_negotiation.cc:407] Connection header received
0923 20:59:25.865946 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.865953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.866067 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.866192 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.867158 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39692: BlockingRecv error: recv got EOF from 127.0.0.1:39692 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260923 20:59:25.938650 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:25.935959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.0.97.61:45791)
0923 20:59:25.936179 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:25.936187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:25.936809 (+ 622us) server_negotiation.cc:407] Connection header received
0923 20:59:25.937048 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:25.937054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:25.937183 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:25.937299 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:25.938398 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39708: BlockingRecv error: recv got EOF from 127.0.0.1:39708 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260923 20:59:26.072136 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:26.070126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.0.97.61:45791)
0923 20:59:26.070310 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:26.070315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:26.070745 (+ 430us) server_negotiation.cc:407] Connection header received
0923 20:59:26.070890 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:26.070894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:26.070990 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:26.071099 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:26.071838 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39722: BlockingRecv error: recv got EOF from 127.0.0.1:39722 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 20:59:26.334012 23299 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:59:26.332152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.97.61:45791)
0923 20:59:26.332410 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:26.332418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:26.332557 (+ 139us) server_negotiation.cc:407] Connection header received
0923 20:59:26.332721 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:26.332726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:26.332799 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:26.332886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:59:26.333719 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39726: BlockingRecv error: recv got EOF from 127.0.0.1:39726 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260923 20:59:26.477562 23415 catalog_manager.cc:1259] Loaded cluster ID: 5bfaa38ed6074ec7aed585364a3ef131
I20260923 20:59:26.477713 23415 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: loading cluster ID for follower catalog manager: success
I20260923 20:59:26.482147 23415 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: acquiring CA information for follower catalog manager: success
I20260923 20:59:26.484315 23415 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3627a5be2307411e87ca6d1fe20b97c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:59:26.563277 23434 catalog_manager.cc:1259] Loaded cluster ID: 5bfaa38ed6074ec7aed585364a3ef131
I20260923 20:59:26.563444 23434 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: loading cluster ID for follower catalog manager: success
I20260923 20:59:26.566502 23434 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: acquiring CA information for follower catalog manager: success
I20260923 20:59:26.568514 23434 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5787a807cf5347c6963ef990e248fe99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:59:26.857666 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:26.851647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.97.61:45791)
0923 20:59:26.852246 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:26.852251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:26.852375 (+ 124us) server_negotiation.cc:407] Connection header received
0923 20:59:26.852552 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:26.852557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:26.852711 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:26.852825 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:26.853320 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.854059 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.855376 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.855784 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.855895 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:26.856461 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:26.857034 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:26.857148 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:26.857335 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 20:59:26.857413 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":294,"threads_started":1}
I20260923 20:59:26.871202 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:26.864066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.0.97.60:44033)
0923 20:59:26.865695 (+ 1629us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:26.865700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:26.865717 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:59:26.865859 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:26.865874 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:26.866060 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:26.866213 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:26.866632 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.867429 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.868749 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.869173 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.869269 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:26.869775 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:26.870452 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:26.870571 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:26.870704 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 20:59:26.870778 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1521,"thread_start_us":1401,"threads_started":1}
I20260923 20:59:26.889395 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:26.877796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.0.97.60:44033)
0923 20:59:26.878579 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:26.878589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:26.882595 (+ 4006us) server_negotiation.cc:407] Connection header received
0923 20:59:26.882901 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:26.882910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:26.883032 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:26.883145 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:26.883990 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.885082 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.887125 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.887816 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.887955 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:26.888491 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:26.888636 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:26.888702 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:26.888949 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 20:59:26.889068 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598}
I20260923 20:59:26.889734 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:26.878019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.0.97.61:45791)
0923 20:59:26.878257 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:26.878266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:26.883469 (+ 5203us) server_negotiation.cc:407] Connection header received
0923 20:59:26.883651 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:26.883661 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:26.883791 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:26.883888 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:26.884481 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.885608 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.887745 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.888284 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.888436 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:26.888859 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:26.888961 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:26.889063 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:26.889208 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 20:59:26.889292 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:59:26.891179 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:59:26.877584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.0.97.62:41895)
0923 20:59:26.884035 (+ 6451us) server_negotiation.cc:206] Beginning negotiation
0923 20:59:26.884044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:59:26.884070 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:59:26.884198 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:59:26.884205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:59:26.884436 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:59:26.884648 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:59:26.885193 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.886184 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.888091 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:59:26.888747 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:59:26.888873 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:59:26.889780 (+ 907us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:59:26.890476 (+ 696us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:59:26.890620 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 20:59:26.890763 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 20:59:26.890854 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6292,"thread_start_us":6176,"threads_started":1}
I20260923 20:59:26.903241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 23153
I20260923 20:59:26.904460 23215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:27.017661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 23153
I20260923 20:59:27.039367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 23227
I20260923 20:59:27.040647 23289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:27.149658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 23227
I20260923 20:59:27.170476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu pid 23302
I20260923 20:59:27.171559 23364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:59:27.278796 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf1GrEz/build/asan/bin/kudu with pid 23302
2026-09-23T20:59:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4981 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9206 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (376245 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST